I think the issue comes down to:
1. We don't have a special term for those who watch movies, read books, or pretty much any other form of similar entertainment. Of course, this sort of comes from the assumption that video games are meant as a storytelling medium, not a competitive arena (such as sports, where such labels abound) or just a simple past time that happens to be a game (where labels like this can exist). Considering the "games are art" would have some issues if we try to label it among the latter categories that allow labels (or at least most people seem to believe so), then going the former route, where we don't have the label, is more popular in some circles.
2. It is a self-identified label, and we shouldn't be allowed to label ourselves (for whatever reason).
3. It causes division, an us vs. them mentality. Considering gamers have a serious issue with inclusion, then adding in division even down to whether or not you are labeled a "gamer" can be problematic.
4. "Gamer" has often had some negative connotations associated with it throughout time, and those who play games likely want to remove themselves from it. You seem lazy, misogynistic, nerdy, etc. regardless of whether or not you actually are, just simply because you choose to identify as part of a culture that many people believe is made up of those kind of people. Of course, judgment really isn't a good reason to remove yourself from a label, but some people have quite the knee-jerk reaction to it because of those negative connotations.
If some of what I said above doesn't already make it obvious, I have absolutely no problem with the term. I will admit that #3 poses some problems, but I think that is more an issue with the culture rather than with the term itself. Athletes don't have anywhere near as serious of an issue with exclusion, at least from what I've seen, regardless of their label. It also is an easy to cover your interest and open up conversation with other people who enjoy video games.