You are right, I guess the way I have always looked at it is that I hate losing to him, but I don't actually hate him. But I get that for some the hate is personal, not just a I hate losing to them type of hate.
I hate the Seahawks and the Patriots, only because they beat the Broncos and are better teams today than the Broncos are. But the hate I have for their teams does not extend to their players as I enjoy watching great players regardless of who they play for.