No difference, just two different attributes. One attribute is age, the other
is gender. In both cases, the outlying attribute provides an obvious and unfair advantage and in both cases it makes no sense to bring the outlying attribute in.
Just as you said physical development with age is real and undeniable, the exact same could be said when comparing a male to a female if we are talking about athletes. Both are cases where it makes no sense to mix the two. It takes just as much dismissal of reality to put a man in a girls sport as it takes to put a 28 yr old in a 10 year old's sport.