If someone actually figured out how many of the starters went on missions, the number is probably lower than 75% (on offense I know that Jamaal, Tejan and Kurtz off the top of my head did not serve missions).
http://www.cougarboard.com/board/message.html?id=16369253
Presumptions (not 100% accurate, but just numbers to make it easy):
75% of players go on a mission
20 players per class
Freshman are 18
Returned Missionaries (RM)
Approximately 15/20 players per class will serve a mission.
Average age for returned missionaries (60 total players) = 21.5
Freshman age: 20
Sophomore age: 21
Junior age: 22
Senior age: 23
Non-Missionaries (NM)
Approximately 5/20 players per class will not serve a mission.
Average age for non missionaries (20 total players) = 19.5
Freshman: 18
Sophomore: 19
Junior: 20
Senior: 21
TOTAL BYU player age (60 RM*21.5) + (20 NM * 19.5) =1680 total years
Average BYU player age (1680 Total age / 80 players) = 21 years old
Presuming the average age of a non-missionary college player is 19.5 years old, the average BYU player is 1.5 years older than the average non BYU player and took 2 years away from the game at a key point in development.
OR The way PGVikingdad calculated it: 2 years x .75 = 1.5 years older