You can watch games with fans of a rival team even if you are constantly getting beat. Them getting excited when they win, or vice versa, is not as hard as you're making it out to be.

Now if they're random strangers, that's a different thing. But if these are people you're related to or are friends with or work with? You either need better friends or you need to toughen up if you can't watch the rivalry game with them.