Simple Rules for Public Discourse: If he said "******", and meant by this that the referee was incompetent due to being homosexual, then he should be derided, run out of town or stoned, I'll leave this up to personal preference. However, if he simply meant, albeit in a coarse and ineloquent way, that the referee was an incompetent myopic moron, then the only person with a right to feel offended is said referee.
The lesson we learn is that meaning is more important than the words you use, complaining about calling someone a 'fag' does not equate with supporting gay rights, merely that you would like others to think you support minority rights. This kind of reaction is just a demonstration that society pretends to be O.K with homosexuality, however, society at large would probably be fine with talking trash about gay people if you used no words that could be construed as having a 'history' of homophobia.
TLDR: Racist, homophobic and sexist sentiments are vicious, unpleasant and hateful, words, however, are not hateful, it's how the words are said that makes them hateful.