This may be good in theory, but won't happen.

Any team not in the top 4 (playoff) will have at least 1 thing that is a question mark on its resume (and that may very well be true for some/all teams in the playoff.) Any non-P5 will always have SOS, whether a legitimate argument or not) and any number of biases against it. And when you compare any team not in the playoff to the team that wins the playoff...whom by definition just beat 2 of the top teams in the country...the winner of the playoff is always going to trump the #5 (or whatever they are ranked) team.

You may get a few holdouts, but the winner of the playoff will get 95% of all AP and Coaches poll.