Of viewers are just casually watching whichever game is on.

If games are all ppv people will only pay for the specific team they are interested in. And a percentage of fans won't even do that.
The amount you would need to charge would be cost prohibitive as well.
I think streaming is the future, but in an ESPN+ type model more than PPV.
Subscribe to College Football streaming Service with lots of games.

If I was ESPN, I would expand ESPN+ to carry more live ESPN, ESPN2, ESPN U broadcasts. Get cord cutters who still want some sports to subscribe and build from there.