Navy is now a part of the AAC (8 conference games), they play Army & Air Force pretty much every year (I think they occasionally may have missed AFA). Plus they play Notre Dame almost every year. That leaves 1 game possible each year and they likely had to do like BYU would have to do if they join a conference and figure out how to extend out and make good on a lot of games currently signed.

Navy had been independent their entire existence in football until recently and they were very full on their schedules before going independent. I'm not sure if they even had easier out clauses like BYU probably went in with all their contracts.

Although it does look like 2020, 2023, 2024 & 2025 don't have that 4th game yet on this site...

http://www.fbschedules.com/ncaa/indep/navy-midshipmen.php

Then again they could have games to fill those spots that this site just doesn't have yet.