And he doesn't seem like a BYU fan, so I very politely asked.
I'm not sure why you feel the need to jump in and defend someone from a politely asked question.
If I had said mean things, or stated definitively that he wasn't a BYU fan, I would understand a little bit. But your reaction seems quite over the top given the circumstances. Which makes me think that you are probably a good friend IRL, or related. No?