I don't think people should feel badly about turning down a calling if the circumstances do not allow it (especially when the SP didn't even know the circumstances, such as in your scenario). Plus, even if you didn't want to turn down a calling, the "technically I didn't extend the calling so don't feel bad" angle seems so weird to me.