the affiliation. With 16 teams and a 3-2-2-2 or 3-6-6 schedule format, you can still play your conference mates often enough to call it a conference.
But, at 18-20 teams, I believe a conference would need the extra game to play your conference mates often enough.
I actually don't mind conference membership models with 18, 20, or even 24 teams if division and conference championships weave into the CFP structure somehow.