Since 1984, BYU has had 4 seasons that were unequivocally successful as defined by 10 wins, beating Utah and a bowl win. 3 of those are under Mendenhall.
In other words, he actually has taken BYU to the brink several times and I think it's much more likely that a coach who has gotten very, very close, even one who is currently in a slump, is far more likely to take a program like BYU forward than a totally unproven entity.
Here's my counter question. What makes you think that taking the football program of a private, parochial school sponsored by a relatively unpopular religion, that has a strict honor code and is based in a small town in the middle of the mountain time zone to the 'next level' is so likely that finding an adequate replacement is likely?
And also, if, somehow, BYU could find a guy that could do as well as Mendenhall (700 winning %), what makes you think he won't bolt for a P5 program the second someone offers?