I don't identify as a "gamer" to begin with. I like many things, and a large number of these things are either old-school thinky-fiddly PC games, or indie art games.
But that doesn't oblige me to care about the idea of "gaming" as if that would be a single thing, and I have no need to "tolerate" all games that I don't like, or their players whom I don't like. I don't care about how many people play games, I'm not compelled to feel glad about the size of the interactive entertainment industry, but I don't feel any particular need to purge "gaming" from them either.
Now, if someone DOES think of themselves as a "hardcore gamer" as importantly as they think of their nationality, gender, or religion, and THEN they claim to hate casual gamers, that's both kind of pathetic, and also dangerously bigoted and hateful.
But if you don't, if you recognize that gaming is just a hobby, then "hating" casuals is nothing more serious than "hating" Twilight fans, hipsters, PC elitists, EA games, truthers, or furries: It's really just voicing your opinion that you don't feel and think like that other group of people, and you don't sympathise with their attitude.