Grounds for losing your recommend would be serious transgressions and/or not fulfilling the standards outlined in the recommend interview.
If you committed a serious transgression, it doesn't matter if you work for the Church or not, it can affect your job. An example would be alcohol abuse and DUI's. If you don't fulfill the standards of the recommend interview, then the Church (employer) does not see you as capable of fulfilling organizational goals. This would be similar to anyone working at a bank who must be covered by surety bonds which could be rejected based on past minor indiscretions.
Too many people who oppose the honor code appear to think that transgressions are merely a mental mistake with no real reason for external consequences of any type.