I went there my first day instead of acclimating, and the hike out of the caldera was pretty tough. I also stayed at high altitude, and had a nasty headache.
I usually try to prepare myself better for the high altitude stuff. Quilatoa is pretty amazing scenery, though.