Part of it is duration, which yes, does and should play a role.

In 1992 Barry Foster ran for 1690 yards for the Pittsburgh Steelers. It was only 29 yards short of Emmitt Smith and still the single-season rushing record for the Pittsburgh Steelers. But Foster isn't in the HoF. Because he never again got that many yards in any other two years combined, and within a few years, looking to replace him, the Steelers landed on Jerome Bettis. The Bus never got 1700 yards in a season but by consistently churning out 1200-1600, he ended up in the HOF.

Beyond that, Romo's peak still wasn't at the same level as Rivers. By weighted AV (where again Rivers is #11), Tony Romo is tied for #232. Ryan is #18 and probably will be a HOFer as well. Stafford isn't close though - tied for #163.

It's also interesting that you want to discount pro bowls since every QB gets them, but Stafford has exactly 1. Matt Ryan has 4. Aaron Rodgers has 9 (helped somewhat by playing in the NFC without Brady and Manning).

In 13 years as a starter, Aaron Rodgers has 201 career AV (15.46). In 13 years as a starter for the Chargers, Rivers has 192 AV (14.77/yr). I'll agree Rodgers is slightly better, but it's closer than you'd think. Rivers deserves to be in.

If there's anyone you should be railing against as compiling stats over a lengthy career and not having a peak, it should be Ben Roethlisberger. Curious how you feel about his case?