Bigger risk is someone breaking in while you are not home and stealing it.
Has happened to a couple of people I know. Never fun to know your gun is out there in the hands of some criminal, hoping you don't get it back from a murder scene...
Just get a safe. Part of the cost of gun ownership if you want peace of mind.