The reality is that you rejected her offer to stay longer to wrap things up. That’s the reality and the narrative.
You feel like you have good reasons for doing it, but look at the optic. And remember that she talks casually with her colleagues and you likely do not.
You have damage control to do.