The formal complaint was made by another organization. A man took the balls and disappeared into a bathroom when he knew he wasn't supposed to tamper. Said man was referred to as "The Deflator". Brady wouldn't give his phone to the NFL but instead destroyed it.

There are probably as many rebuttals on the Patriots part, but that isn't the issue. The issue is that the judge, jury and executioner ruled on the facts he was given. Case closed. You can't negotiate that into your contract and then try to flip flop when you don't like the ruling.