When you refer to "casual gaming" and presumably casual gamers, do you mean people who play casual games à la Farmville, Bejeweled and such, or do you mean people who play mainstream games casually, i.e. sporadically and for fun, without regard to score or efficiency? There's a huge, important difference between the two.
That said, I will assume you mean the Facebook gaming crowd.
First, I don't think there's a large contingent of casual gamers who go on to become mainstream gamers. For the most part, casual games are so different from mainstream games (casual games are mostly puzzle- or task-based, while mainstream games are mostly immersive and scenario-based) that there aren't really that many reasons for casual gamers to jump into mainstream games. It's hard to imagine how to say Bejeweled is childish - at worst it isn't interesting, at best it's a test of skill - and Farmville is hardly an immersive or imaginative experience, but Mass Effect can quickly get demoted to "Playing space captain like a 6 year-old boy" in the eyes of the non-gamer.
Sure, they might spend more time playing FarmVille than it takes to finish all of BioWare's games, but the stigma that surrounds mainstream gaming, the difference between casual and mainstream games, and the expense of mainstream games are all barriers to making that leap. So some gamers might see these casual gamers as cop-outs (I personally don't, however), or people who are co-opting gaming and making it more socially acceptable to game while also changing what it means to be a gamer. Some may make the leap, but how many 40 year-old women have gone from FarmVille to Dawn of War, or from Bejeweled to Dragon Age?
Second, the fact that people in the news media often conflate casual gaming with video games as a whole means that a space friendly to gaming is being constructed in the public mind, but it's a space that is centered around casual games - which means "mainstream" games will still be at the fringe of what is socially acceptable. Naturally, as gamers, we want people to both accept gaming in general as a pass-time, *and* to accept mainstream game genres (RTS, RPG, FPS, MMO, etc.) as being a valid pass-time. If society accepts games under the assumption that "games" means what we'd call "casual games", then we've made zero progress.
As it stands, "gaming" in the broadest sense may be becoming accepted, but non-gamers at the same time are beginning to see casual games as the "normal" games and *our* games as "fringe" or "hardcore" games. Us gamers who prefer intensive titles will continue to be at the fringe of the socially acceptable, which, as someone who wishes our interests were as mainstream as those of people who watch movies or television, is irritating.
I have nothing against casual games, not in the least. Why should I? But I don't like the media acting as though all games are the same thing. "Mainstream" games are narrative, artistic, simulated experience that immerse you into some scenario or another. Casual games are digitized board games with neat concepts. Both have their place, but to act as though they're all the same thing is to lose what makes each kind of game special.