It's a University. People not of the church can go to school there. It's operation as an institution does not much resemble the way the church runs.
And on top of that, most LDS BYU haters are haters because of BYU sports. Sports is even a level further removed from the Church.
If BYU stopped playing sports tomorrow, how many haters do you think would be left that are LDS?