So what if it's a minority of any group opposing a distasteful name or other hurtful practice? Should we ignore them because they're the minority?
A brief look at American history reveals that that kind of thinking has resulted in some of the worst laws and periods in our history. From slavery to Jim Crow laws to the treatment of Mormons, when the majority gets to dictate to the minority what is or is not good for them, the result is not often good.