There are different ways to say this without disrespecting their academic tradition and value.
You could just say "our RPI will take a hit", or refer to strength of schedule.
This conference has been very gracious to invite us. No need to disrespect them.