I think a fandom can ruin my interest in a piece of media if the fans are loud and numerous enough to make the developers start changing things for the worse. Like, for a vague example, there is a series of some sorts about a small group of people with several main characters, one of whom is very popular with the fan base but I'm not particularily crazy about. If the creators of said team see how popular that character is and make the series almost exclusively about said character, that would make me lose interest completely. Or if a particular element of a story, like a romantic subplot or a fighting style or something, becomes popular and then the creators decided to axe everything else that I liked about the story to make room for that element, it's the same thing.
So yeah, I think a fanbase can definitely make me dislike a franchise I have interest in, and they can certainly turn me off from trying to get into a new franchise as well.