If someone's actively doing something that bothers you, I'd say it's okay to ask them to stop, but thinking less of them just because they do something different than you is more questionable. That said, I don't think minorities should have some sort of immunity to any and all criticisms, even ones that have no connection to their status as a minority, simply because they're met with actual ignorant bigotry some of the time. Judge people by their individual actions, not their demographic's general image and if something they consciously do bothers you, politely ask them to stop. It's like hanging up on a telemarketer, you can't hold the fact that they do what they do against them, but you can still say "Sorry, not interested" and hang up the phone. When they keep pushing the issue after you say you're not interested however, that's something you can get bugged about without worrying about being intolerant.