(Insert group here) privilege is getting a better set of assumptions assigned to you as the default.
If you were in a conservative Muslim country, and a Muslim accused you of dishonesty, the majority there will likely believe him over you; i.e., you do not start out equal. The current of his society is continually manufactured to flow in the direction that would benefit him, so all he has to do is coast, while you have an upstream swim ahead of you if you want to be believed in his stead. He is given a privilege that you are not.
The "privilege" doesn't mean the colloquial "privileged" of possessing material wealth. Nobody is placing a dinar in his pocket over yours. Nobody is showering him with diamonds. But he still has something that you do not. And his privilege of the favorable assumption while you take the unfavorable one can have real consequences.
So, in your example of the homeless white guy? Well, what if he knifes a homeless black guy? Says, "Self defense. Stood my ground." Question is, is he treated the exact same way as a homeless black man would who stabbed a white homeless guy and said the exact same thing? Or would the white guy get the benefit of the default perspective -- you imagine yourself in his shoes against the "other" in both instances, so the black guy is always on the out? He is either the criminal that you (empathizing with the white perspective) are stabbing, or the criminal whose attack that you are fending off? And is that the majority perspective? Will you find that time, and time, and time again?
Have you seen A Time to Kill? It has a scene which tends to highlight that, even when you think you're sympathizing with a black --when you think you're fully in their shoes -- you're not. You're not giving them the same range you would a white. Your conclusions are distant and generic -- you don't ever swap them into your shoes, and so you never grant them the privilege of the reaction that you grant yourself or someone you do identify with. Without that reaction -- without starting them out as you -- you deny them the downstream path. They are on the outside, and your "I'm treating them 'equal'" is not equal, because you are not treating those on the inside the same way. The unenlightened emotional processing never notices the difference because it's checking whether the conclusions don't follow from the premises, not whether they're using different premises. The check for bias only checks what comes after, sees no forcing in the logic there, and so comes out "All clear!" -- completely missing where the bias resides.
And the thing is, this bias comes about naturally. If you don't realize it's there and counter it, you're going to treat people unequally. Racism isn't just putting on a white hood and going around saying, "I hate blacks." It can pervade society as a continual, persistent perspective.
Oh, and the easiest way to spot someone who is racist in this way is to just listen for, "But minorities can be racist, too!" The perspective is straight up "I'm in my shoes and they are the 'other'." Such a person is placing themselves at the center of the universe and they are only looking at what what might affect them. If they had ever entered the other pair of shoes -- really entered as themselves and truly flipped the status quo -- they'd have noticed that minorities do not have the power to force on society the default assumption that they are the good guys. That's a rather big point. So while it is true that an individual minority may be rude to you for no reason but your race, you still have the bulk of society granting you white privilege. His racism is a single breath that does nothing to reverse the flow of all the benefits you are granted just for being white, while your racism adds to the river against him. He has nothing backing him but himself, while you have every Conservative Talk believer ready to back your whiteness.
Your racism has societal power. That's some white privilege.