Especially those regarding race. The time, location and audience are critical. He was trying to carefully hold the country together at the time.
Also, his “send them back to Africa” comment was actually a voluntary recolonization. Only if they wanted to go.