I don't know that I would say that fanservice bothers me personally, or at least not to a large extent. That said, I do feel that it speaks to the general perception of the immaturity of anime fans. Now don't get me wrong here, if I were to say that I loved anime, it'd be a massive understatement. That said, I think that for people who aren't familiar with the conventions of anime, fanservice is one of those things that reduces anime's credibility as a genuine artistic medium. One of the ways that I always defend anime is by pointing out that it, much more so than western cartooning, is used to tell serious stories with legitimate depth. Fanservice doesn't add anything to that depth, but it does turn people off from considering that depth in the first place from the medium as a whole (again, don't get me wrong here, I definately understand that not all anime series are meant to be deep and meaningful, but some are, for instance Ghost in the Shell or Kino's Journey), and I think it's a shame that this, admitedly immature aspect of anime, is allowed to bring down the credibility of the genre as a whole.
I realize it's not going away, so I'm not going to go getting my metaphorical panties in a bunch over it, but I definately understand why it turns people off to anime, I can't say that I feel that it really adds anything of value, and in my opinion anime would better as a whole without it.