This is the problem with medicine and testing.

First step in pediatric workup for fever, "is there a source." Your newborn probably didn't have one, so then you go down this road of blood cultures and further testing. I would guess your son was also irritable, crying and inconsolable.

If clinically he doesn't have other symptoms (which basically means he is drinking and peeing ok at that age) one could argue a period of observation might also be a reasonable approach.

Unfortunately our medical environment often dictates exactly what happened here, a $5K workup for nothing.