I know that this has been discussed ad nauseam, so I'll be brief. My frustration with the defense is not that they lose games; it's that the defensive scheme is, by nature, fundamentally incapable of winning games.
Long, high play-count drives tire an offense out and increase the likelihood of a possession-ending offensive mistake (fumble/INT/penalty); but this strategy also tires the defense out and, as the game wears on, increases the chances of a defensive mistake. When the game is on the line, it turns out that opponents are very good at running short and intermediate routes to the sideline, and - even worse - we now have gassed corners trying to make tackles in space and exhausted defensive linemen trying to get pressure.
BYU relies on the offense to win the games - and that works, right up until it doesn't. We've seen enough with this defense to know that when BYU plays a competent team and needs a late-game stop, they can't depend on getting it. When the offense has a bad night, the defense just isn't built or schemed to bail them out.
BYU's offense lost the Boise State game, no question - there were so many turnovers and failures to convert that I don't see how the fault could lie anywhere else. But the defense was incapable of bailing out the offense, and ultimately gave up a 71-yd FG drive that sealed the victory for Boise.
When the team's success depends on the offense always having a good game, fans can be justified in thinking that the defense is the weak link overall.