Generally violence should be avoided. But yeah if they were deliberately coming over and over to cause you problems it just might be justified. Of course you'd have to weigh the consequences, and consider your specific situation, but sometimes a good pop to the nose will solve a situation better than anything else.

Now I can't answer your hypothetical question for sure, but I can say there is a place in this world for violence, and my religion undeniably teaches it.