"my" team has a shot to make an impact on the national stage. If they're going to consolidate all the possibilities with just a group or two of the chosen invitees, then why would I keep paying near the attention?
If I just want to watch great football I can just tune in to NFL. I imagine lots and lots of other college football fans might feel the same. Seems there's the potential to lose a lot of fans who today watch the big schools, but maybe tomorrow will not.