If it happened as you just described, play got physical, you got tired of it and held up your fists while threatening him, he grabbed your arms in an effort to protect himself, you land a punch and he then decks you in the face...you were 100% in the wrong in that scenario.