they can draw.
It's not a stretch to think some athletes aren't attracted to a culture that is 99% LDS, 90%+ white, 30% married (or whatever it is), and drinking, sex, and facial hair are banned.
Lifestyle doesn't negatively impact athleticism, it just narrows the pool dramatically.