If you grew up in a wealthy family your options are probably pretty good, even if you were lazy in high school. If you became a ski instructor for 10 years, and then decided to buckle down and shoot for med school, I bet your parents would probably still support you (this is exactly what one of my friends did).
If you come from a poor family, you don't have those same advantages. Do you think it's just as easy for a kid growing up below the poverty line, in a high-crime neighborhood while being raised by a single mother, to get straight A's and high SAT as it is for an upper-middle class suburban kid, with two university-educated parents including a stay at home mom who's making sure he finishes his homework every night? And do you think that affirmative action then makes it so the first kid is now at an overall advantage over the second one?
The people I see complaining about how unfair and "completely racist" affirmative action is are usually the ones who already had a whole lot of advantages to begin with.