This was the roster of playing time in 2003 where the wheels fell off against one of the toughest schedules in BYU history:

3 quarterbacks: Berry, Beck, and Mortensen. True Freshman Beck was a Lavell pre-mission commit. Jackson Brown ended up finishing the season due to injuries to the remaining QBs.
3 running backs: Whalen injures his foot in pre-game warm ups before the first game. New recruits Brathwaite and Fui carry the load. Tahi was back from a mission, but played fullback instead.
2 tight ends: Crowton found true freshman Coats and Justin Jory to carry the load.
3 wide receivers: Toby Christensen, Chris Hale, and Rod Wilkerson was the primary tandem.
The Oline needed to be rebuilt around freshmen Jake Kuresa, Ofa, and Eddie Keele. Vincent Hanale stepped in for Scott Jackson because of an MCL injury

On defense, the 10th leading tackler was a holdover from the Lavell era (Ifo Pili). The remaining players enrolled with Crowton or later.

It's clear from this that the talent level needed to be restocked. Nobody could win with that roster. Crowton found the likes of Pitta, Watkins, George, Collie, Harline, etc. before being let go after the 2004 season. Crowton did the right thing and the team won about the right number of games based on the talent level at that time.