Yeah, I took a long break after episode 5 of season 4, for the same reason.
It was weird that I liked the series and yet it somehow felt like it was dragging along. The last half of season 4 is better. I watched those 5 episodes over the course of 3 days, and I enjoyed it.