The schedule is very difficult even without factoring in the travel which is crazy.
The fanbase expects 10 wins annually.
The buy in from admin is lukewarm. Money for coaches, recruiting, and facilities is limited.
Getting kids into school is very difficult. Grades, Honor Code, are extreme inhibitors.
In short BYU schedules like a P5(with worse travel), spends like a G5, and has to recruit like an Ivy League.
Not a great recipe for success.