That part makes sense to me, as long as you did leave a buffer originally.
I can understand someone else showing up in your group late, and then everybody having to scoot down a seat. I agree that is a little different than just going up and sitting right next to somebody in an unfilled stadium.