You want refs to either not notice you or respect you.
Apologize and then watch your tongue in the games. It may take a couple games for him to believe you and stop holding a grudge, assuming he does.
Were I a ref, I would pointedly not give the benefit of the call to any loudmouths or whiners.