It's not as small as one of the academies, but it's much smaller than nearly any other school, P5 or G5. You can only take kids who are religious (they don't have to be LDS, but they do need to be religious), are willing to live the honor code (a tough sell for most college athletes, even LDS ones), and (if not LDS) are willing to be surrounded by Mormons all the time.
That's a small pool. Then you have Utah, who has been more successful the past 20 years and who is a member of a P5 conference, fishing in the same pool, cutting it down a little bit more.
I agree with the OP, a football team composed entirely of the top LDS talent would be a very good team. They might struggle at cornerback and maybe running back, but they'd be good to great pretty much everywhere else, and you can win a lot of games with a great pass rush and zone coverage if you have a good passing offense that can put up points.