I have found myself frustrated by the Defense in games where we didn't give up many points. Likewise, I have found myself thinking that the Offense played as well or better than the defense but realizing that we didn't score enough.

The NCAA Rankings seem to bear this out.

NCAA Ranking Total Offense BYU 40th vs Scoring Offense BYU 71st.
NCAA Ranking Total Defense BYU 62nd vs Scoring Defense 48th.
Source https://www.ncaa.com/stats/football/fbs/current/team/27

I think our emotions get wrapped up in the play to play of the game. If the defense is surrendering yards but finally getting stops, it doesn't feel like the defense is playing well. Conversely, when the offense is moving the ball well but not scoring enough, it feels like they are doing good things just needing to work on redzone play.

On closer inspection though, it is the points that really matter. BUT I don't think I can get over the play to play failings. I still feel like the defense is worse than the offense.