It would require at least 7-8 teams to leave. It is an 8 team minimum to be a conference and the left overs would need to also have landing spots to keep the league from adding more.
I just don’t see the chatter between that many teams and conferences to be kept secret.