We can argue until we are blue in the face about whether or not fans are being silly for feeling that way.
But I think it is pretty obviously reality that fans were more willing to attend a game against CSU, a conference opponent, than they are against UConn, a non-conference mid-major that is arguably a "bigger name."
It is what it is.