As men, we are often weird that we crave relationships, but don't want to seem too interested in relationships. We develop relationships "side by side" rather than "face to face", so probably he feels awkward going by himself without kids to be focusing on while having side conversations with his brothers in the ward.
I get it, because I think we all feel like that. We use our kids as a bit of a shield sometimes to make social encounters less awkward. Probably that's something we should all work to get better at so we can still develop friendships after our kids have all moved out.