Do it. Our son has been in a gifted program since 3rd grade. He's now in 7th.
He's been challenged, had many more worthwhile field trips than my other children, and is better prepared, I believe, for the rigors of junior high/high school. We're hoping to get him on track for graduating high school with his associate's degree.