When people say, "It's only a game", when complaining about people hating games for irrational reasons, I think that, if games are going to progess as art, they have to deal with the shit without people babishly defending them with that phrase.
But, when someone tells me to not take games seriously, usually in competition or co-op, I tell them that, if it "only a game", that they're making quite a big deal about me taking it too seriously. I once told this person in a multi-player match this, after I started getting a bit frustrated over being sniped the kabillionth time by the same 2-people (honestly, who wouldn't get frustrated about that). I wasn't cussing or yelling into my headphone or anything - I didn't even have a headset. I just typed in chat, "I keep getting killed by the damn snipers", or something along those lines, and some guy, probably in high-school, said, "Ha! Dude, it's only a game, stop getting worked up about it!". I said that I play competetively (which is why it's called a competetive match), and it's just a habit. He didn't respond, at all, and left the lobby after saying, "Dude, you take things way too seriously", and left, which was weird, because he was there long before I had joined - he had played a couple of matches, and ending abruptly like that led me to beleive that he didn't want to start an argument, which wouldn't happen. Who was taking games too seriously, then?
Really, the semi-existant morale of the story is that, no matter what, we as humans will always take things seriously, or act competetively. So, I don't like it when someone says it, really. It just sort of annoys me.