The counter argument is why shouldn't I be able to create a community where everyone wants to have the same standards and so we all sign a contract promising to do so. That's what an HOA is.
It does get complicated sometimes, but in theory it makes a lot of sense that an HOA has authority.