If you are a W2 employee, that can be a problem. If you have earned income over about $16,000, then for every $2 over, I believe SS is decreased by $1. I understand that at 66 (or 67), this withheld SS is returned to the taxpayer, but I've not looked into it.
If you are self employed, like me, then it just takes some income management.