Come November... it's BYU who, when Conference play is wrapping up and there are games that are deciding conference fates. We are far less significant in November, than any current conference. That is absolutely a direct result of Independence. The same applies to Notre Dame, unless they are in for a playoff hunt.
College football, like it or not, functions based on conference play first and foremost. Independence is like a trailer before a movie.... the second that movie starts... we are forgotten, we are irrelevant... we are less relevant than 2 team playing for the Sun Belt Conference championship.
So yes, Independence is absolutely a problem, because it's not how College Football currently functions, or NCAA sports in general. It's about winning your conference, doing enough to make it to the post season and then winning whatever championship format is set up. In FBS... it's winning your conference, doing enough to make it to the post season and play either for a National Championship playoff, a New Years day bowl... or just playing for a bowl and winning it.
Independence = complete irrelevance.