My son refs and the league has told him that he has authority to kick out parents...
and the league has made it clear to parents and coaches that those decisions get respected or else. It has made it so coaches regulate pretty well and keep parents in check. They usually speak up well before the ref gets agitated.