Admittedly, that's an easy mindset in which to find oneself in feeling that, but
it's simply not the case, and safety record of Teslas and other auto-drivers (while still the vast minority) on the road bear that out.
Now, where I'll agree COMPLETELY with that sentiment is in the fact that if it WERE an all-cars changeover, the advancements and efficiency would be mind-blowing.
This is an older video, but the simulation it shows is real-life capable.