Though I would contend that two bottom teams squaring off shouldn't be disregarded.
You are probably right though, most of the argument would be comparing middle teams, when discussing how tough it is to play in x conference vs y conference.
In some ways, the bottom dwellers should have the most separating them. That is where you probably see the largest difference in money, visibility, name recognition. Having no conference to promote, what else is a school like Temple left with to get kids there?
The "Conference Comparison" game has no rules, so no real answer to the question. Either way it makes me chuckle when a Vandy loses to a Temple.