points do matter to the voters. They cannot all be insiders on every team so they are going to base their votes on perception as well as wins of course. So who and how a team wins definitely makes a difference.

Barely beating an unranked team at home when ranked 15 gives many voters the impression that, BYU may not be as good as advertised. If they win and teams ahead of them lose they will still move up, unless someone behind them wins better, aka with more style points, then they may get jumped. Especially as they move closer to the top ten. This is especially important for BYU as they may not be perceived as strong as some of the other teams in the top ten.

Unfortunately its the world we live in, with the whole voting system to rank teams. Perception is going to play a role. Until the CFP expands to include more teams and ideally a conf champion from every D1 conf. We are always going to have somewhat of a guessing game going on and a big part of that is perception.