Here are your P5 opponents since 2011:
2011: Miss (2-10), Tex (8-5), Utah (8-5), Ore St (3-9)
2012: Wash St (3-9), Utah (5-7), Ore St (9-4), N Dame (12-1), Ga Tech (7-7)
2013: UVa (2-10), Tex (8-5), Utah (5-7), Ga Tech (7-6), Wisc (9-4), N Dame (9-4), UWa (9-4)
2014: Tex (6-7), UVa (5-7), Cal (5-7)
2015: Neb (6-7), UCLA (8-5), Mich (10-4), Mizz (5-7), Utah (10-3)
2016: Ariz (3-9), Utah (9-3), UCLA (4-8), WVa (10-3), Mich St (3-9), Miss St (6-7)
So that's 30 P5 opponents. That's 12 "no-bowl teams" (40%), 9 "unranked" teams (30%), and 9 "ranked" teams (30%).
This means that at you don't tend to play "average or better" P5 teams. It means you tend to play "average or WORSE" P5 teams. 70% of your opponents fell under "average or worse".