For almost every other FBS school, their first four games are usually mostly OOC, and most schools only schedule one or two tough OOC games. We are counted as a tough OOC game, so the odds early in the season will always be that the opponent will have just played a cupcake before facing us.
Sucks, but that's the life of independent BYU.