1) The Church/school dynamic is probably pretty tough. You still go to church with fans of the program. I'm sure wives and kids don't enjoy hearing how their peers think their spouses/fathers are 'complete idiots' or 'suck at their jobs' or 'need to be fired'. Because of the church aspect, there is probably a little higher fan investment.
2) The job may be harder because some coaches may take on the role of 'ecclesiastical leader' — or one that tries to keep them away from 'worldly distractions' that wouldn't be an issue elsewhere. This shouldn't be an issue for most players (I was a student and the HC was the way I lived anyway), but as some recent players have shown, booze, drugs and the ladies are a difficult tempation.
3) Not being in a predictable conference is difficult. This doesn't mean that going back to the MWC is what I want, but playing familiar teams each season and having a conference title to compete for CAN BE very helpful. Independence is great, but the lack of familiarity CAN BE difficult for game-planning and motivation players....
4) BYU missed out on several players because of academics (i.e. Star L. - ended up a uu). It plays a role with some players, but not all.
5) BYU fans have been spoiled. BYU has seen great successes. It can be hard to maintain. If you slip up (see 2017) and some fans will NEVER forgive you — no matter what improvements are made.