It wasn't just Jews, it was Muslims, gays, Communists, and blacks
He was more against any race or trait that he believed was undesirable. The Aryan race was seen as the ideal but he essentially was looking for any light skinned people because he believed them to be genetically inclined to be the hardest workers and most trustworthy.