What are you using to 'hand buff'? Any polishing compound is basically sandpaper..
So even by hand, you are 'sanding' your car with fine-grit sandpaper.
It's better to do a paint correction once every blue moon... then do everything you can to protect the paint (ceramic coatings + no-touch carwash or proper hand wash).