You can have a very similar argument with the term "RPG"
You get ( an alarming amount of) people saying "Well EVERY game is an RPG, because Roleplaying game means you're playing a role"
Well, sure you're playing the 'role' of Chell in Portal, but I've never seen anyone call Portal an RPG series (not seriously at any rate).
The term may often be vague or confusing, but almost everyone accepts that games like Final Fantasy, Disgaea, Kingdom Hearts, Mass Effect, The Elder Scrolls, Chrono Trigger and Diablo are RPGs. While games like Call of Duty, Mario, Kirby, Tekken, Silent Hill, Need for Speed and Tetris are obviously not.