Louisville produced Johnny Unitas. Other than that, Louisville honestly has no football history to speak of prior to the mid-90s. Louisville also has its share of natural disadvantages; for starters, it's the second school in a poor state. It was also traditionally a commuter school specifically tasked with educating Kentucky's "urban" (read: poor and black) population. And its fan base is relatively small and highly localized.
Despite all that, the school and the city basically decided 20 years ago to compete at the highest levels of college sports, and since then, they've made it happen. They've invested enormous amounts in facilities and now have perhaps the nicest facilities in the nation. They've hired great coaches. They've built thousands of dorms and converted their campus into a gorgeous residential space that draws in students from all over who love their teams. They win in every single sport. Their fan base is small but very passionate; Louisville is a metro area with over a million people, and none of them care about professional sports.
The ACC was very candid when it invited Louisville. They admitted that they looked around at which available school had the best overall athletic program. Louisville did, so Louisville got the invite. Props to those folks for making it happen. They earned the spot they currently have.