But since he had no way of getting a hold of you and that was your fault, if I was in this situation I would just pay and move on.

If the wrong number wasn’t left I would say it was almost entirely his fault and while I wouldn’t refuse to pay I would explain my rationale and maybe offer to pay half of what he was going to charge.