of faith in Peter Jackson, mostly because they trusted the source material. At the time, Jackson had mostly made small budget horror films.
Most studios won't commit to shooting a trilogy until they see how much money the first film makes.
Look at what happened with Eragon or Artemis Fowl. Both movies came from popular source material that appeals to young audiences (though not nearly as high quality as Tolkein's world-building).
But for various reasons (Eragon because the film was terrible, Artemis Fowl because they had trouble getting it off the ground and then the delayed release unfortunately pushed it back into the pandemic), the movie didn't (in the case of Eragon) or won't (in the case of Artemis Fowl) make enough money to justify sequels.
Ender's Game is another example.
Obviously, with Star Wars (as with Marvel), there would have been a big enough guarantee to make more, so they maybe could have done it there.