First, over filling an automatic transmission is a known hazard. It will ruin the transmission quickly. Even a "shade tree", DIY type mechanic is aware of the problem. So this sounds like a (frankly) bone-headed error. It is *POSSIBLE* that whatever problem you thought you were fixing was fatal and the fluid change only accelerated an imminent demise. That's about the only saving-grace I can imagine for BIL or the shop.

Second . . . the shop should carry insurance for accidents that cause damage to customers vehicles. I've not had to analyze this specific issue, but CGL policies exclude "your work" and generally cover accidents that cause damage. This feels like that. BUT, I wonder if this meets the deductible even. Shop probably has to pay something out of pocket.

Lastly - assuming the error is as clear as it looks, shouldn't your BIL have the same worry you are having? That he's taken from you (or clearly owes you) money and deciding how to deal with it could ruin the relationship? Does he know?

While I live in a world of blame and liability, I do think some things are simply accidents. Mistakes. Not the result of culpability. If you can get your mind to that spot, you might feel differently about who should pay for this. Doing something like splitting the cost could feel reasonable if you see it that way.