When you signal, you expect a few things. When someone violates that and gets right up in front of you where the ball is coming down and their head or anything else is in the way for just a second, that affects it. That's just one problem. Not many people on here have been punt returners before but that's a pretty obvious thing to know. It has nothing to do with being touched.
I'm notr sure you know what "fair" in fair catch means.