That can't be good. Is there any law he is violating by living so close to you? Aren't there laws about not living by a school? Anything?
I would make sure that he knows that you see him as a threat and if he even THINKS about your family he will be in a world of hurt.