Obviously the fans weren't trying to offend someone. But it certainly broadcasts their ingnorance and lack of understanding. And for that, I am embarrassed for them. So, maybe the best response for the OP to have used would be, "Hey dude... It's not that I'm offended by what you just said, but when you use that word, it makes everyone around you embarrassed for you. It makes our fanbase look bad. Do yourself a favor and don't use that word."

I'm not going to lie, whenever I hear someone use that word, I think in my head, "That dude is the true definition of retard." But I usually don't say anything.