That's fairly normal. Similarly, men never seem to be subject of sexism.
First of all, I believe it's because of power. White people are more powerful, they matter more.
So do men. It's like how when you call a kid names as a grownup makes you mean, but when he calls you names, you have to...