I agree, and have a ND source (admittedly not a ridiculously well-connected one)
that says from everything he's heard, that's the ND admin's exact line of thinking.
Which is why it's all about ND's access to whatever playoff or National Championship exists. If that goes away without them joining a conference, they'll join.