This is part of your problem. Obviously YOU didn't think it was racist, or you likely wouldn't have said it, or at least not worded it quite like you did. Intention or not, jokes or not...you have to consider the 'optics/aurals' of your sentence and wording. Say it aloud. If that doesn't sound racist to you, then you are likely the minority with your opinion (and I don't mean just on these boards).
It doesn't work to say something brash and then put it on others as, "well their reaction is on them, because *I'M* not the racist...". Adding the 'you' preceding the "black people" comment is what makes it sound bad, imho. It's directing and targeting/singling an individual or group. I know it was your reply to HIS comment in-kind and intended as humor, but you have to acknowledge that in text with no tone and no body language, things often get mistaken and intentions twisted.
Just hoping to help you see it from the other side, so to speak...as silly or undeserved as you might feel it is...
Would sound the same as you stating, "...us white people are _______." versus someone of color saying it as, "YOU white people are ________." It just sounds bad and accusatory.