The key switch happened way before that. Adar tells the tavern guy he has a job for him right before the Numenorean's arrive. They wouldn't have known he was an enemy so he would have been able to get away.

I think they're setting Halbrand up for a tragic downfall as one of the Nine, rather than a betrayal. We'll see.