I would stop talking to him in games and focus more on your kids and coaching
them.
It sounds like he is in your head and is a big distraction. It sounds like you have a good team. I've noticed that the refs are going to call the game tougher on to best team in the league. Its not fair but thats just how it is.