If you aren't the best team in your conference then you aren't the best overall.
Until the CFP expands to more than 4 teams it should absolutely be limited to only conference champions. If you want to put in teams that didn't win their conference then it needs to be expanded.