It is such a regular occurrence that I expect it. I live in Utah County. I am a Ute and I often wear Ute gear on game days. I get comments almost identical to the one made to you all the time. Like the comment made to you, they are clearly joking around. I typically respond with some attempt at humor and even though I am not very funny we all give a courtesy laugh.
I guess we all see things a little differently.