Looking at the numbers, I just don't think that's completely true. With the exception of about 15 teams, viewership depends mostly on channel, time, and game stakes. If you've got Utah and Wazzu (for example) playing for a conference championship (no playoff implications) on ESPN at 4 PM, that's no different than Arizona St and Stanford playing for the same stakes at the same time.

Or Iowa St and Baylor. Or Kansas St and TCU. Or so forth.

If the stakes are higher, it obviously helps, but there's only a few teams that really deliver beyond what a generic team would bring in.

Alabama, Georgia, Florida, LSU, Texas, Oklahoma, Clemson, Notre Dame, Ohio St, Michigan, maybe Wisconsin, maybe Nebraska, maybe Penn St, USC, Oregon, maybe Washington. Probably missing a few others, but other than that, they're just generic teams in roles and timeslots.