The part I've quoted is what makes you not racist. Unfortunately, the original poster did not give this caveat, and the guy in question never said it. When you act like a jerk, like that guy did, people are more likely to not give you the benefit of the doubt in things like this.
And, note...