A word can't offend you unless you let it. One of my very good friends back home in Baton Rouge is black, and he would always jokingly refer to himself as 'the Black'. Eventually it became a running gag between the two of us (and somewhere along the line we started using heavy Russian accents, making it seem even funnier). He embraced the slightly racist joke, so if anyone would try to use it in an offensive manner it would have already lost all its power.
Anyway, I like to think like Yahtzee on this matter, e.g. if there's no malicious intent behind the joke, then it's all okay. If you're telling a ****** joke while wearing a pointy white hood and lighting a cross afire...well, that's probably crossing the line...