The other parts are year-round sunshine, altitude, and culture. As has been stated in this thread, many to most don't care about the sport. Perhaps if swimming was more nationally prominent, they would. Perhaps if BYU spent more and expected more from the program, they would be more nationally prominent. And on and on.

National level swimmers go to universities where swimming is a priority. And where is it warm, and where the altitude allows more quality and more volume of the work required to get to the level they want to be on.