The final poll of the year prior. This at least gives value to strong finishes, and it could be argued that many of the things that made those teams successful are still there.
Sure, players graduate, coaches move on, etc. But it’s already been shown that the preseason poll is a random guess anyway.
But where you start impacts where you finish (to a degree), in that it COULD cap your ceiling (for a lot of teams that aren’t blue bloods).
So, start with the end of last year and go from there. Teams that are pretenders will fall down the ranks rapidly. And those that keep winning will stay at the top, just like they did in the year prior.
The next best option is to not rank teams for a certain number of weeks. But that will never happen, so here we are.