The biggest detractor to going independent for most teams is financial. The way most conferences share revenue, the biggest teams at the top of a conference end up subsidizing the smaller teams to some degree. The gap is wider in G5 conferences where the teams at the bottom of those leagues are usually worth next to nothing in media rights. So, most teams actually make more money being in a conference than if they struck out on their own.

The second reason is that it's a royal pain in the yewt to make independence work. You have to find an acceptable home for your non-FB sports, work on your own to schedule three times as many games every year as everybody else, figure out your own bowl arrangements, generally travel a lot more, and a whole list of other quirks.