How about losing 9 straight to Utah? How about losing to garbage teams like Toledo and USF year in and year out for the last few years? How about not having a consistent football program and coaching? I would say all of those are much, much worse for BYU than Utah making into the college football playoff.
Could it have a negative impact on BYU? Sure, but it's no where in the same ballpark as some of the other major issues and problems BYU currently has. It definitely isn't close to being the worse thing that could happen to BYU.