All human beings are racist, from birth, it's genetic and natural. Even infants will stare longer at pictures of faces from people of their own race, etc.
Some adults work harder at fighting this innate tendency than others, some are more successful than others, some have decided it's not worth fighting at all.
Even the most ardent progressive white people will almost never be found living in areas with high concentrations of blacks for instance.
What they advocate for society often is very different from what they practice in their own personal lives and who they choose to associate with.
As for members of minority groups, it's really rather easy for them to avoid being called racist because firstly, all they have to do is use "diversity" and "multiculturalism" as code words for their racism, for the time being, because they are effectively the same at this time. They advocate more diversity, the end result is more power for their group. And they got to do it without having to specifically say it in a way that might have sounded racist.
However, the more important thing shielding them from being called racist by most people is just the mere fact that they are a minority, which means that usually even if they do speak in very blatantly racist terms they will get a pass for it. White liberals can only see racism in other whites. They either cannot recognize it, or refuse to acknowledge it, no matter how virulent, when it is in a minority person or group.