I'm an out of shape middle-aged guy who wants to start swimming for fitness. How big do you think the pool needs to be to make lap swimming worth it (for you)?

We have a pool in the backyard but its certainly not a lap pool. Just curious where you swimmers would draw the line in terms of it being worth it / not worth it.