I work in a hospital, and I see people wearing masks touching the outsides, fiddling with the mask, etc, and just spreading whatever contaminants are on the mask to their hands and then other surfaces.
Washing your hands and not touching your face is more effective than mask wearing. Note: that is not the same as saying that masks are not effective.