If a person thinks of himself or herself as above average in something, saying they're average is a slam. But someone who believes themselves to be far below average would consider being rated as "average" as a compliment.

And I would expand this to include how that person is generally thought of within the group or groups he/she exists in. "Average" should be compared to how the group generally thinks about the person or thing being rated.