I agree with JamesDean's post. I live in a predominantly Active LDS community. Our residents in many ways fit many of the LDS stereotypes with the exception that my community is overrun with young families, who are largely not BYU fans. Many of these young families attended the U at a time school spirit was at an all time high and a time where BYU football struggled to rise above mediocrity. Many of today's youth in Utah are aware of Utahs's '04 run, '08 run, and their move to the PAC 12. BYU has not had a "special" season in a way that captures the attention of the youth like Utah has in quite some time...
I think this year having a dynamic duo with Taysom and Jamaal and a good season could start a swing of the pendulum, in fact I think its starting. That's the vibe I get in Utah at least.
Back to the stadium issue. I hear over and over from Utah residents what a pain it is to drive through Utah county on gameday and then the drive home. Whether we all agree on this or not, I hear so often that it becomes an all day event because of traffic and parking plus the game time of course. Utah county was poorly designed for the amount of people who live and travel through there.
If BYU continues its winning ways, the buzz will continue to build and the seats will continue to fill. Until then the last few years of mediocrity coupled with the amount of time and effort it is to attend the games has evidently hurt attendance.