They aren't going to be left behind, they aren't going to sit still in a lame duck conference without SoCal access. They aren't going to be able to recoup the losses of USC/UCLA with any addition of schools.
If schools believe that the departure of Washington/Oregon is imminent or a future certainty, why would they leave one conference that is relatively stable for the other? Especially if they can stay home and make the same or more?