Wow, there are a LOT of misconceptions about race, racism and what makes someone a racist.
Racism: ?noun
1. a belief or doctrine that inherent differences among the various human races determine cultural or individual achievement, usually involving the idea that one's own race is superior and has the right to rule others.
2. a policy, system of government, etc., based upon or fostering such a doctrine; discrimination.
3. hatred or intolerance of another race or other races.
2 isn't really relevant here, 1 and 3 are what we're talking about. Clearly you can make a racist joke or use an offensive word and that doesn't make you racist. It can at times make you a jerk dependent on the circumstances but the beliefs behind it are what would make you racist. Note however that intent to offend isn't required, if you believe that one group is more likely to commit crime, or be alcoholic, or like watermelon you are racist. Also worth noting is...
Race: 1. a group of persons related by common descent or heredity.
5. any people united by common history, language, cultural traits, etc.: the Dutch race.
There are a lot of definitions for race but these are the two most relevant here, just to point out that race can be used to define cultural or even religious differences.
So far that the OP, I think that your friend misunderstood your joke, which is unfortunate. You offended him and owe him an explanation and apology but that doesn't make you racist. To the several people that make racist jokes and use racial epitaphs you may or may not be racist, but you ARE perpetuating racism. People don't need to stop making a fuss to end racism, people need to stop perpetuating racist stereotypes and stop placing wedges between people by using offensive language. For those who say that this or that is just a word that's true, but racist language carries with it not only it's own offense but every stereotype that goes with it, if someone gets offended because you said it that's because YOU did something wrong, not them.