When you back in, it is into a clear and marked space with few hazards.
When you back out you do so into active pedestrian and vehicle traffic.
It is only confusing to.... Well I can't think of anyone it should be confusing to. It is a common practice that many people do... Why would it confuse anyone?