I remember this was one of the criticisms of Brady Quinn
And for good reason. You can't really find an example of an established NFL QB who's also chiseled.
Even the elite QBs will do some physical prep work (see Peyton below), but I agree with you, you can only spend a certain amount of time preparing and every minute spent lifting in the gym is a minute that's not spent preparing in some other way that's more important for a QB to be preparing.