I think the main reason I'm not big on it is twofold:
First, it's essentially Spaced done wrong. In Spaced all the jokes, references and nerd-culture was subtle. It served the plot and served the characters. It was never just a reference in a vacuum. In TBBT, the references are often apropos of nothing. In essence, the fact that they referenced something is often meant to be taken as a joke. That isn't funny. That's just Scary Movie as a sitcom, if slightly less mind-numbing.
Second, and this is just me being petty, the culture that's risen about it. I know I shouldn't get pissed off about this but the whole "gosh, I'm such a nerd" thing is fucking irritating. I didn't start reading comics and playing D&D because a wildly popular sitcom includes both as reference points. I started because I enjoy them. I'm not here to be cool I'm here to have a good time. But hey, as far as I know I've actually just invented this type of fan and they're as real as the "gamer girl" stereotype we've been yelling about lately.
But I wouldn't say I hate the show. Just dislike it.