His insinuations could have been based in two things: 1) Brady's politics (which Stephen A later admitted he didn't know what they were), or 2) Obama's race.
When asked what he was insinuating by Skip, Stephen A. responded, "I'm not insinuating anything, I'm just saying". I've heard him use that sentence 100 times when backpedaling from a ridiculous insinuation - "I'm just saying".