1. They often come against teams that we shouldn't lose to. This is the biggest factor in my emotional state after a loss. The Utah State loss had me down for about a week, not just the weekend. You should never lose to Utah State. An expected loss to a better team really doesn't affect me that much.
2. Every bad loss hurts our street cred. It would be different if we were in a good conference, but we need wins more than most teams because we're going it along. It's gotten to the point where losses involve not only direct sadness from the loss itself, but also worry about how this loss is going to affect BYU's future in relation to the P5. You might call this stupid when reading it in print, but for an emotion-driven fan it's a reality.
I think a lot of this angst comes from not having any sort of safety net. We'd be a lot more comfortable as a fanbase in general if we weren't by ourselves, still trying to fight for legitimacy in the changing landscape of what defines a relevant team.