would it have been a flagrant for pulling him down? I realize that is not what happened.
Just because you miss the ball when you foul someone doesn't make it a flagrant. What about someone going strong for a dunk and you hit them on the wrist and they end up falling to the ground? Probably depends on the situation, but often it is just a common foul.