At the college level every move they make is scrutinized. They are given a copy of the film and an evaluation by their governing body. If the ref receives poor evaluations they will find themselves out of a job the next season.
While I agree the refs sometimes have poor games the conspiracy theories about officials purposely screwing us over are stupid. These people have too much time and effort invested in their training to screw things up because of a petty bias.