He calls cops. Do everything you can to notify him that although you haven't used the access much in the past, you plan to do so regularly from now on.
I assume you are not locked out. If so, coordinate with him to daisy chain your lock with his. If you need to research this more , BM me. I can help.