Many, if not most, derogatory racial terms are derived from descriptors to skin colors. The "N" word is derived from Negro--black in most latin-based languages. It may have at one time been self-referential or not carried the kind of connotation that it later picked up, but it has since become a terribly negative, incredibly loaded term.
In that same vein, Redskin, while maybe a self-referential term in the 1800s, has become outdated and largely negative (if not terribly offensive term like the "N" word). Can you imagine any other mascots being created as a racial term? Not group-based, like Aztecs, Utes, Seminoles, Fightin' Irish, etc., but actually race-based?
Let's just take out the negative race-based terms (and you all know or have heard them--I won't repeat them here). Even the neutral terms would be largely unthinkable. The Washington Caucasians? The San Francisco Pacific Islanders? The Houston Hispanics? The Green Bay Blacks? No way.
It's time to move on. It's not worth offending a large swath of people (even if the "majority" don't see a problem with it) to maintain a name and mascot that has marginally positive roots and history.