Not just that. There's nothing in the canon that says Sauron didn't try (and fail) earlier or that Sauron had no existing relationship with Numenor before he tried to tempt them with the rings of power.
I stand by my statement that there's a good chance Halbrand is Sauron. We'll see.