I suspect in the new series HBO is doing, we'll get better answers to this. It seems that the Night King almost went into a period of "hibernation" — though that isn't the right word. After the last war with the Night King, the one that ended in a draw, the Night King fell back and went almost dormant (though we know he was taking Craster's — and likely other — male babies). During this time, the wildlings were free to live their own lives without any serious Night King threat.

Of course, over the last period the Night King had woken up, which was the reason for the Wildlings to gather together under Mance Rayder and for them to try to get South of the wall.