If anything, make a small gluten free version of things. No way I'm missing rolls, pies, etc just because somebody else doesn't want gluten.
I don't eat stuffing, but I don't forbid everybody else from having it.
It's not like a peanut allergy where they can't even be around gluten.