He gave a lot of of talks, and finding the exact quote is proving difficult, but I remember him saying in a conference address once that while building a "comfortable" home for your family is fine, going beyond that was wasteful and the money spent could and should be put to better uses.