If you crunch the numbers down only 0.5% of the white population in America owned slaves in the 1800's. That small percent owned thousands of slaves but still, 99.5% of white americans had nothing to do with slavery other than dying to protect that institution when the Civil war came around [or dying to remove it]..And those who did fight and die to protect slavery were the dirt poor redneck whites who were lucky to own a house let alone own a slave. State's rights was the lie to get the poor southern whites to fight without knowing it was another word to fight in the name of slavery.
The sources I've read indicate that there were approximately 350,000 slave owners, as per the 1850 Census..
However, when you consider the fact that slaves were inherited and remained property of the slaveholder's family after his death, then that number can easily be quintupled or sextupled...
And then factor in the amount of indirect beneficiaries, people who's livelihood or income revolved around slavery such as slave overseers, lawyers, merchants etcetera, the number is increased even more..
All of this is irrelevant though, because as Woolfe and SheHateMe said, the overall culture at the time was permissive, and supportive towards slavery.
Slavery cannot exist in a culture that condemns it (at least not openly), which is why the War eventually occurred..