This year we played a schedule that could charitably be described as 'hot garbage' (kudos to Holmoe for putting even that together), and we finished #11 and a yard away from a NY6 appearance.
Honestly, "only" playing 4 or 5 P5s isn't the worst for our reputation. Over the last ten years, BYU has a 40% chance of beating any one P5. Obviously that's a top-level number, and it varies depending on the quality of BYU and of the P5, but it means that if we play 7 in a season, odds are we lose. If there's 4, there's a better chance to upset the one high-level option, beat the weak option and pull out wins against the two mid-level.
Don't get me wrong, I enjoy more big games, but if we're looking for high-level ranking/success, 4 or so is probably the sweet spot. Enough to prove ourselves without additional risk.