Because he knows he would become the most hated man to come out of BYU
Everything he did for BYU would be erased in the minds of BYU fans. Wins over Texas? Nothing without loyalty. Going to Michigan is fine, they're not BYU's rival. Going to Utah is unacceptable.