I would never take a risk at that age. 4 years ago our son (1 month old) got a high fever from who-knows-what, and we spent Dec 23-25th at Primary Children's Hospital. Luckily he was able to come home at 9:00 am Christmas morning, but that was something I never want to go through again.
Relatives "wishes" are FAR less important than a newborn's health. The crowds alone (all the coughing, sneezing, etc) would keep me away.