When you compare teams like TCU or Utah (quality teams ranked high) to teams like Texas A&M or Oklahoma (mediocre at best, but with huge followings), you'll see those bigger-name programs still get better slots.
So it's about on-field success but also about program prestige.