In the 2005 LV Bowl, Cal had a backup (Steve Levy). I think he did start a game or two before the bowl. It helped he was going up against BYU's weak defense with skill players like Marshawn Lynch and DeSean Jackson.

Actually, Air Force's backup quarterback came in later in the game against BYU in 2005 and torched the defense.

UCLA had a backup quarterback (McLeod Bethel-Thompson) in the 2007 LV Bowl. They almost won, but it was a low-scoring game and his stats weren't impressive. I don't recall, but I think both he and Levy may have even been third string in the seasons they faced BYU.

Ole Miss switched to Zac Stoudt at some point in the 2011 opener against BYU. The starter, Barry Brunetti, only threw three passes, though. Also, Ole Miss had suspended a quarterback, who was competing for the starting job, for that game. Stoudt seemed to spark the team a bit, but didn't exactly light BYU up and obviously had the costly fumble that gave BYU the victory.

Hawaii lost its quarterback before the 2011 game against BYU. The warriors had to start a backup, but also switched quarterbacks at some point in the game. They had mixed success, but not enough to stop BYU from winning comfortably.

Georgia Tech switched quarterbacks against BYU in 2012. If I recall correctly, they brought in Vad Lee, but his passing and rushing stats were meager. He was the starter against BYU in Provo the following year.

I'm sure there have been other backups to play against BYU, whether the whole game or coming in at some point (not counting late meaningless drives when the game was already decided), but I think we've covered all the noteworthy games.