So that's a bit defensible. If you had BYU at #16 on your ballot last week (as some did) and then they lost, yeah, they drop into the 20-23 range, maybe even out of the rankings.
That said, there's also folks who had BYU #6 or #7 so they'll probably drop BYU less