The vast majority of people watching a game on TV or from the stands have a pretty shallow understanding of the rules and what is or isn't a foul or violation in most circumstances.

I've reffed HS varsity ball for 10+ years. I'd watched and followed basketball for my entire life when I first started out, but when I started getting trained and studying the rule book, I found that I knew surprisingly little about even some of the very basic rules. Even years later, I'm still seeing different plays and scenarios every season that make me go back to the rule book and get advice from more experienced guys.

You can't trust fan judgment on these things. It's just not accurate. And that's not even tapping into the fact that fans are hopelessly biased and actually see very little of a game in a way that they can really analyze what a referee is doing and how well they're doing it.