They are probably the biggest factor. Kw said it a few weeks ago.
Can a lower ranked team win a single game? Sure.
What would be interesting is if someone compiled recreating averages and compared them.
So take byu vs team x. Average the previous 4 years ranking on 247 for each of the teams. And see how closely it correlates to the winner of the game.
Would also be interesting to compare to final sagarin rankings. Would tell you how good the coaching staffs are.
I don't know if anyone has done that, but they should.