I agree that it isn't "fair" to compare somebody from a good home environment in a nice neighborhood to somebody from a very difficult home environment in a bad neighborhood.
But what is the "merit" we are trying to measure? Is it "grit," such that we should adjust for the degree of difficulty?
And, I am pretty sure, it has been shown that lowering admissions standards in an attempt to adjust for difficult circumstances doesn't result in the students from difficult circumstances doing well. In fact, for law schools, they often don't graduate, or graduate at the bottom of their class.
Lastly. . . race is a poor analog for "difficult circumstances." Poor white kids don't get the benefit of affirmative action, despite coming from exactly the same situation as poor minority kids. To those poor white kids, it really seems like they are punished because of their race.