Have him contact all his buddies. Let them know that this is a football-only party. If they wish to bring their wives or girl friends, fine, but the only entertainment being provided is watching the game. If they chose NOT to watch the game, they are going to be bored.

Your main obligation as hostess for a football party is tend to the people watching football. If some people there decide to split off into their own little side-group away from the game; then as a good hostess you really should check in on them every once in awhile to make sure they are comfortable, well-supplied with munchies and other refreshments, and perhaps engage in a little friendly chit-chat for a minute or two, but that is all.

The party's entertainment is watching football. If they chose to wander off by themselves, that's their problem, not yours. You cannot be expected to change your carefully planned entertainment simply because some guests decide they don't wish to participate in the activities you've scheduled.