issue. Actually, it's kind of a weird "loophole" in LDS thought, if you really think about it. Many other religious groups that take the Sabbath very seriously don't make an exception for travel. For example, 7th Day Adventists or Conservative & Orthodox Jews.
One of the officers in my company is an observant Jew (but doesn't wear a yamulke or a beard of anything like that), and he does not travel or commute between sundown on Friday and sundown on Saturday. He walks out of big meetings in order to make sure he can drive home before the sun sets on Fridays. When he travels overseas, he mananges his schedule very carefully so that he is not traveling anywhere during the Jewish sabbath, on either end of his trip (this was tricky when he would visit us in China). He builds some serious cushion into his travel & commuting schedules to make sure that he is not caught traveling when he shouldn't be. I'm not surprised that non-Mormons might assume that observant Mormons have a similar approach to sabbath travel, given our hard line on Sunday play.