Because when you compare casual gamers to mainstream gamers, casual gamers usually don't bother to get very acquainted with the industry they are buying from, and usually are people whom have made the mainstream gamer's life hell in the past (Mario and like is still something, to them, that only the "geeks" do, while the "cool" people only play things like FB games and iPhone apps).
So, people can be "casual" gamers, but I wouldn't personally call them "gamers" until they at least have some knowledge about the medium as a whole they are tiptoeing in. Ah, and actually play games that take more effort than just logging in every morning to check a FB game status.