Do you know when he does this? Go stand in front of your door with a hot drink
like you're enjoying the morning next time you see him. Strike up a conversation with him so he sees you standing in front of your door. Maybe he'll catch on and point it in another direction