If you just switched Utah and BYU on their schedules, Utah's would be tougher
So I am not sure I would say "significantly more difficult". You play UW and USC - so do we. You play Tennessee, Boise and Utah State. We play Washington State, Arizona, ASU, Cal, UCLA.
Your best opponent is currently Utah. That helps your schedule.