but so you can use those folks as a reference.

Every time I've switched jobs they've asked for references, including a direct manager. People understand they can't ask for your current direct manager, but they will want your last one. If you burned bridges by trashing him or her then you won't be able to get that reference. Most industries are pretty small so even if you won't need a reference, you still don't want a bunch of enemies out there.

And even when you are leaving a company, it is perfectly fine to do so without burning bridges because people understand that life chances. My last employer wasn't happy when I told them and tried to talk me out of it, but ultimately the story we spun together was that I was going to a smaller company but with much more leadership responsibility, to "Try things out on my own." If it doesn't work in 2 years, everyone will have warm feelings about each other.

Frankly, I would use the standard of "would you tell that person the critique to their face."
If you would, then go ahead and say it. If not, it's probably better to just keep it to yourself.