You're right but there's more. There are BYU fans all over here. Certainly more than the number of Ute fans you'd find right around Provo. BUT, you don't see the Y fans. Why? Because we're tired of the judgment and negativity that comes with it. At rival game time, sure, I'll wear my full on BYU gear. But the rest of the time, I want to get along with my neighbors, and sadly, that doesn't happen when I boldly declare my allegiance. I suspect that's the case with many of us here. We have to be "closet fans" because of the negativity around here. Salt Lake is a ridiculously polarized society in some ways. I have to caveat that I live in the Avenues (Liberal Central) and attend church on campus, so my view is skewed. But I've learned my lesson - don't put up a BYU flag unless you're prepared to be egged (true story).
Ps. I've lived in SL for 10 years, and 5 in Provo prior to that.