I don't think there would have been anything wrong to kindly point out that you found the term offensive and why. That gives the people who said it the opportunity to learn and grow. Then of course they'll still have the choice whether to continue saying it or not, but at least hopefully then they will be better informed.

I have a nephew with down syndrome, and this is the approach his parents take. I think, generally, people are not looking to offend others with what they say and are merely ignorant of the hurt behind the word.

Though the words are not equal in power, what if they were yelling that the refs were the n-word? I think the r-word holds similar, if not a completely equal sting to those who have loved ones with down syndrome.