We actually did high tea at Butchart Gardens and not the Empress Hotel. The gardens have a great patio for serving tea/food which overlooks the flowers. That's were I would take them for tea. We also explored some of the corners of the Empress Hotel - sure reminded me of The Shining.
Take a water taxi from one side of the bay to the other. Great to watch the sea planes land and take-off.
If you ever go with just your spouse, I know the perfect spot for an awesome getaway.