http://en.wikipedia.org/wiki/Utah_County,_Utah
I know fans come from all over the State and country to attend games, but let's assume that the local fans are your base.
There are more than twice as many people that live in just Utah County alone in 2014, than did in the mid 80s - the glory years of BYU football.
Yet, somehow we still can't fill the stadium. I'm not sure what to make of that. Certainly, there are a lot of factors - people choosing their 60" screen over the live event, for starters.
Also, 80s BYU football was REALLY fun to watch - throwing for 400 or 500 yds per game - I think that has a lot to do with it. GFGH is fine, but it's not a big play offense.