The hate wouldn't bother me. They already hate us and always have. But better chance at a bid (i think) way more entertaining rivalries and better competition on average.
I would feel bad leaving the WCC because they have been a life saver and welcomed us with open arms, but outside of the top three that conference is terrible.