#5 and BYU were #6, BYU would be left out. In other words, if one of the following teams is #5 then #6 BYU gets left out:

Army
Liberty
New Mexico State
UMass

If it's another P5 team, they take their conference's contracted NY6 spot. If it's the G5 team, they take the G5 autobid. BYU just needs the at-large spot. At #5 they're guaranteed. At #6 they're virtually guaranteed (other than if the above 4 teams are #5).

Even at #7 they're only left out if the #5 and #6 teams are in the same conference / both G5s (or again if one of those teams is one of the above 4 teams).