Academy Football just doesn't scale up well to P5 levels. Navy and Army both have large markets (both where they are located and military members) and have certain advantages that give them huge viewership (Army/Navy game, Notre Dame games) and nobody has invited them still. Because they just don't work at a higher level.