Many scientists say that a reduced caloric intake stimulates metabolic pathways that help to keep us alive. Like the body's defense mechanism during times of starvation and near starvation. And the thinking is these metabolic pathways evolved during the many thousands of years humans lived on the brink of starvation in a hunter-gatherer existence. And today's relative age of gluttony has suppressed these defensive life-sustaining mechanisms.
Maybe sperm count and its life proliferative qualities rides along with these other pathways.