I don’t know why it’s a problem. You own the property, you decide what to do
with it. Unless you live in an HOA that restricts it, or have other restrictions for the property that were in place at the time of build, government needs to stay the heck out of private property ownership.