Earlier in the movie he was speaking with Mattie about a previous gunfight he was in where he and his Navy Sixes were outnumbered:

“You go for a man hard enough and fast enough, he don’t have time to think about how many’s with him – he thinks about himself, and how he might get clear of that wrath that’s about to set down on him.”

Like Shimmy said, I'd like to think he was headed for the hills or back to his family. Gang life no longer suited him.