OK, trying to answer without getting into too much personal detail, I'd say that it dawned on me that hearing negative words used for white people and hearing or reading extended negative comment on white people as a group (one that sticks in my mind was reading of the NOI's scientifically-novel theory about white people all being descended from experiments with cross-breeding humans with dogs, but there were far more mundane and plausible examples) might have been disconcerting or annoying or upsetting at the time they happened...but that I could rapidly forget all about them because I could just put the book down or stop listening to that person and go right back into my white-majority world.
Whereas my non-white relatives didn't seem to have that same luxury (including my own dad, who got called racist terms fairly often, whereas I never did) because they were obliged to exist in a world where such negative attitudes were all over the place and had real power behind them.
I'm honestly not at all sure at this remove if it was specifically about family, or if I had the same response about racial minority school friends, etc, but logically the awareness of relatives must have come first. And to be clear, I don't pretend to share any of my ethnic-minority relatives' culture, I've always identified as 'white' (or maybe 'Londoner', which to my mind implies at least some element of mixture!) and anything critical said about 'white people' definitely includes me.
To be fair, I have some similarity being that my dad was Jewish, although non-practicing, and never taught me a lick about Judaism, and I had very little exposure to Jewish people growing up. To boot, I don't look particularly Jewish and my last name is uncommon enough that most Jews don't recognize its origins. But this mixed identification/some degree of witness of antisemitism are still (probably significantly smaller) parts of my identity.
Still, I seem to observe many people that don't have these mixed identifications exhibiting the same beliefs and behaviors, and having those beliefs and behaviors generally supported as moral by majority peer groups.
I'd like to posit that racism* is borne in part from a person feeling threats to their power and seizing opportunity to classify others in a way that reaffirms their power. Thus, if we are generally good people, it is easy to empathize with this behavior in those who have legitimate and improper threats to their power. And those who possess a power imbalance are unworthy of empathy because it is hypocritical to believe they need the power they seek.
*definitionally troublesome, but collaborating on a precise definition could be specious
There are some people who are at least significantly sociopathic and/or sadistic who would be racist with conscious intent and enjoyment of harming others. But my experience has taught me that these people are very rare, even among those who behave very badly. Many more people have learned the behavior through specific instruction or necessity as a tool to ward off feelings of powerlessness. Often, the behaviors are repeated outside of a direct threat to someone's power, but that's mostly because we function autonomously in the ways we are accustomed to. The behaviors would have been learned as a tool to face legitimate threats to power.
But that's true for the disadvantaged as well as the advantaged. It's the way the human mind works, so why is there no room to appreciate the humanity in someone who acts this way from an advantaged position?
This, I suppose, is where the reasonable person will say that the advantaged person doesn't
need to be racist to protect threats to there power, therefore their action is bad. Well, I agree, but only under the condition that the advantage person logically and emotionally recognizes this, which I think is not such a reasonable expectation of people.
Harry Harlow did an experiment (more complicated than this summary) where he took infant rhesus monkeys and gave them a choice between two crudely created wire monkey figures. One with no clothes carrying a bottle of food. And one with cloth but no food. The monkeys overwhelmingly clung to the mother with clothes but no actual sustenance provided. Decades of attachment research has followed, and although controversial, the general idea is that human beings are not really much different from rhesus monkeys except in our capacity to believe we are different.
My point is, someone's actual power, wealth, nourishment, physical capability, etc. are very poorly correlated with our psychology. Someone may be born with wealth, physical gifts, health, intelligence, large social networks, and every thing you can think of that might make you jealous. But if they were raised by a nanny in an environment that took exquisite and attentive care of their physical needs but deprived them of attachment, it is quite possible they have suffered far more in life than someone who was raised in poverty, congenitally deaf, a victim of repeated and significant traumatic racism, but whose early caregivers provided for their every emotional need and supported their process of separation.
In short, I think the ways we measure whether someone has been born into a fortunate situation in life are in severe conflict with reality. We might do better to appreciate that.