It could be argued, but I personally don't think it's a valid point. The fact is, gaming has evolved as an art form.
Consider film. Anyone can enjoy a film like The Matrix, but it takes someone with patience and appreciation for movies to enjoy a more dramatic or artistic movie like Pan's Labyrinth or Casablanca. But that doesn't mean The Matrix is somehow dumbed down, or simple: it's a rather complicated movie with heavy philosophical and ethical questions posed within a complex narrative and impressive action scenes. But these people don't understand that: they are not the movie buffs, they are not the film students, they are simply people who watch movies for entertainment's sake. They watch The Matrix for simple reasons, but that does not make the movie simple: rather, the person has a simplistic appreciation for movies.
People who don't see the artistic value in a given art form will enjoy it for rather shallow reasons. One could play Call of Duty with appreciation for the intense atmosphere and strategic freedom, or they could play it because they get to shoot things. One could play Halo: CE in appreciation of the story and the mysterious atmosphere, or they could just skip all the cutscenes and blow crap up.
It's not that first-person shooters are dumbing down gamers: they are not somehow an "inferior genre" or anything. Rather, they are gaming's "action" genre: anyone can enjoy them, but only the true appreciators of the art can really appreciate them. And those who play casually will not appreciate games like Mass Effect or Final Fantasy, because they are not enough to hold their attention.
We have to understand that not everyone appreciates games like we do. It's their loss, really, but it upsets me as well. Point is, it is not the fault of the genre, it's simply the lack of true appreciation for gaming on the part of the players.
P.S. I also understand the side of the competitive gamer, the one who appreciates games as a sport. Know that I am not referring to you, but rather to those who play only recreationally, with no true appreciation for sport or storytelling.