I just happen to think that smacking someone in the face is typically a foul - and telling players that it isn't now may not end well.
Going back to the call - they said it wasn't a foul because the contact was marginal. I didn't find the contact all that marginal. I mean - he didn't get a tooth knocked out or anything. But it was enough contact to stop anyone from playing - which to me means it wasn't marginal.