I would absolutely go and not worry about it. If the baby is bundled up and stays warm, then you will be fine. It sounds like to me that you're just an overprotective parent which I totally understand. I would not worry about it, however. The baby will be fine as long as he stays warm.
Congratulations by the way!