And let's not forget that if this just a case of racism, here's Abraham Lincoln quotes:
"If I could save the Union without freeing any slave I would do it...."
"I never have been in favor of bringing about the equality of the white and black races."
"whatever negroes can be got to do as soldiers, leaves just so much less for white soldiers to do..."
And here are Jefferson Davis'(president of the Confederacy) beliefs:
"On the economic side of slavery, Davis fully believed that slavery was a temporary necessity in order to develop the cotton trade in the South as New England's textile industry at that time depended on southern cotton.
In spite of this dependency on Black slave labor, Jefferson believed that gradual emancipation, at some time in the future, would come for the Africans and that they should be prepared for this responsibility called freedom."
He also taught his slaves to read and write, and hold their own courts to decide punishments.
So yeah...