There is an auto spot for the highest ranked G5 team. BYU is not a G5 and therefore cannot compete for an auto bid. They would have to beat out other teams for the remaining at-large spots. Technically, they need to be in the top 12 of the final rankings to be considered.

There are six bowls, for 12 spots. 4 spots are taken by the CFP teams. The winner of each P5 conference is taken in the CFP and/or the CFP.

2019 - G5 was Memphis, no other non P5
2018 - G5 was UCF, no other non P5
2017 - G5 was UCF, no other non P5
2016 - G5 was Western Michigan, no other non P5
2015 - G5 was Houston, no other non P5
2014 - G5 was Boise, no other non P5

So, in all, twelve schools are selected for these major, top tier bowls. These are required to include the champions of the "Power Five" conferences (ACC, Big Ten, Big 12, Pac-12, and SEC). In addition, the highest-ranked champion from the "Group of Five" conferences (The American, Conference USA, MAC, Mountain West, and Sun Belt) is guaranteed a berth if the group's top team is not in the playoff.

In order for BYU to realistically make it, they will need to be more attractive than the number 2 team of the SEC or the ACC or the PAC or the B12 or the B10. So, probably BYU will need to figure out how to make their schedule a little stronger, make sure it's in the top 12 final ranking, and make sure it looks more attractive than the second place team from any one of the P5 conferences - oh, all that and also not have another G5 that meets all those same criteria, like an undefeated UCF and an undefeated SDSU or something similar.

Again - to improve the odds, BYU probably needs to see if there are some good quality games still available, they need to win big as often as they can, they need to go undefeated, and they need the hypothetical second place teams from power conferences to not look so great, and they need only one G5 team to undefeated. All possible. Just a lot of "if's" in there in an already unusual college football year.