from late April through early June (basically, about the first 6 weeks after BYU graduation)

The first 6 weeks of fall (from about September 21 to Halloween)

The holiday season (effectively, the last 6 weeks of the year)

For me, the holiday season makes the list, not so much because of the weather, but because I get a lot of time off from work and, in general, I think Utah is an above-average place to be during this time of year (for example, I always hope for a white Christmas and northern Utah has better odds of this than most places).

The worst times of the year are almost a tie between January-March and July-August. January-March has inversions and potentially snowy roads. But there's at least some variability in the weather to keep things somewhat interesting. July-August is not only too hot and dry but it's also boring (same weather every day). Like you say, summer in Utah is only nice if you go up into the mountains.