different sports more than they used to. In my case college football changed a lot from the rivalries and bowl games that I loved and I just dont find the frequent SEC obsession of the national media that interesting, why would that be unusual then not to be as big of a fan of the game? I also used to love hunting and now while I enjoy it casually I have new hobbies that have replaced it. Same with pro basketball - used to love it, now dont even watch a regular season game and only a few NBA playoff games. Honestly, I think many, many people change interests over time and the favorite hobbies of youth arent the same as those when an adult.
If you think otherwise we'll agree to disagree but I'd point out that 30-40 years ago horse racing, boxing and baseball (remember it used to be referred to as "Americas pastime") where huge, perhaps 3 of the top 4 major sports. Those sports are at this point largely not relevant unless it is a rare marquee event. Those fans who were serious followers of those sports certainly went somewhere, so if you believe that its rare that people dont change I'd suggest baseball, boxing and horse racing declines all suggest I am right and is it not uncommon as you suggest.