politely say you overheard them say a word that is offensive and you have a daughter with Down syndrome. You could even say “normally I just let things like this go but, respectfully, some of us are offended by that word.”
He either is more mindful of it in the future or he ignores you but I think it’s fair to say something when someone uses a word out of ignorance.