crime-ridden neighborhood in a single-parent home where they never really knew if they were going to eat each day is going to have lower scores and it's not because they were just lazy. If you had to constantly worry about your personal safety and hunger, school would seem very unimportant.
Often, people in those circumstances are also part of a racial minority. It's not a perfect correlation, but it is a strong one.