I've definitely "worked" harder, in both physical labor and in an office setting (two very different settings with two very different definitions of "hard work") than I did on my mission.

The thing that made the mission great was having an eye single to the glory of God. It made things WAY less stressful for me. It was HIS work, not mine. I literally forgot the world and just worked. It was long hours, and there were hard times, but it didn't stress me out nearly as much as my career has, and was never even close to the physical labor I put in growing up on a farm/ranch.