I know a handful of people (gisl included) that either never served or came home early.
Great people in my mind, trying hard to do what is right and without going into details, I can tell you life is not as easy in the church for them as those who did serve.

Some of them were told they couldn't have a certain calling cause they haven't served and other crazy things.
I personally heard a Bishop tell a guy that things would have been so much better for him if he had completed his mission.

This bishop may be right, but you don;t need to tell someone that, they already know and are fighting hard to combat what they feel about not being "the perfect mormon".

Many people in the church don't care whether you served or not, but lets not kid ourselves, there are many who look down on someone who didn't serve or didn't complete a mission.