...a ball in contact with a referee is considered the same as in contact with whatever point on the floor the ref is standing at.

I'm assuming the referee in your situation scooped the ball up as it was about to roll out of bounds so he/she wouldn't have to go chase it. It's a little lazy on their part (and a real ref would make the players gather the ball anyway), but that's what you get with intramurals.