money out of retirement accounts to pay off my home, but there is something reassuring that if things got bad we could live on relatively low amounts of money. People will say to invest it due to high rates of return and if that makes you happy do that. It is a relatively low rate of return to pay off the mortgage but the peace of mind for me was well worth the lower returns.