I think it might be related to why there's a rip on the wave of so-called 'anti-intellectualism' lately. Let me break it down for you.
You don't see fine art being celebrated on the television. You don't see weird, Lynchian-inspired auteur films hitting mainstream cinema and scoring big at the box office. You don't see people recite Shakespeare and poetry by Miur on the nine o' clock news.
You see 'backwards' shows like Two and a Half Men and My Name is Earl. You see another sensationalist news story about something that probably didn't deserve to be reported on in the first place. You see films like The Expendables and The A-Team. One dimensional comedy shows appealing to stereotypes (the promiscuous male, the lovable rednecks, the trailer-trash ex-wife) and brainless action flicks. You see a story about an autistic kid who got labeled a cheater by microsoft(i read somewhere it had been reported on Fox news, i think that's the American news network), rather than the next great poem by our generation's young Tennyson or Keats.
Casual games versus 'core' games is, essentially, the same. We're losing our Wuthering Heights and our Charge of the Light Brigade for, dare i say, the video game equivalent of the Twilight franchise. As games become less about story, characterisation, artistic design and 'auteur-ism' they end up becoming noving more than mainstream drivel, degenrating into such games as Farmville. I think a lot of core gamers fear this, and so they defend their core games to the death in the hopes our Bioshocks will never be dumbed down into the likes of Kinect Adventures. Casual gaming, is, in essence, the 'anti-intellecutalism of the gaming industry'. Simple, derivative games aimed at no particular audience except for the ones that can pick up and play within the space of five minutes, as opposed to a 40-hour epic where you learn about characters and their struggles through careful interaction and dialogue.
'Casual is bad' is simply so because of the fear that it may change the industry - indefinitely. As companies see the far more lucrative markets of housewives, children and the elderly as opposed to the niche core market, they may shift their business model to cater primarily, and eventually, solely to that demographic, the core audience will feel betrayed, hurt and confused that they have lost something that primarily belonged to 'them' as a culture that only they identified with.