Some people have never been to a basketball game. The most common location for the rowdiest fans is behind the opponents basket. That is where they are the loudest and have the most effect on the opposing team.
But by all means... don't bother keeping silent just because you don't have the experience to understand. I don't think anyone will think less of you for it...