The first roughing call was not because he hit Taysom a bit late, rather that after the hit he wrapped Taysom up by the legs and tackled him a second or two later. If he would have just hit Taysom and stopped, nothing would have been called.
On the second one, it was because he hit Taysom in the head -- while it seemed like a rather weak hit, hitting the QB in the head should always draw a flag.