Would you rather work for the ACC or the American? The SEC or Conference USA?

If you represent one of the current P5 conferences, you have a vested interest as a referee that the teams in your employing conference hold sway and your conference maintains a strong position. This is in a vacuum, you flat out have an interest in your employing conference doing well.

Think about your own job. Are you interested in seeing your company/employer succeed? What if litigation is brought against it for some business practice at a distant branch in another city. And you're the arbiter.

Conflict of interest. It may be small, but it exists as a matter of fact. Almost everything in life can be explained by motivation and self interest. Fortunately, most refs don't seem crazy biased, but I think the "human nature" or bad decisions are easily swayed at times.