I've got more UI stuff coming that should address that. Your base star rating will be based on your overall star to post ratio. I can't really work out the exact ratios until I see some of the statistics. I don't want it to be too easy to lose or gain stars, and will be removing the ignore score from the mix eventually too.

Regarding how visible they are: that is intentional. They get more visible the more stars the post has. Five star posts are very easy to see, while 0 star posts are almost invisible. Before, it was too hard to easily find the extra-star posts, and this makes them stand out more. It also makes the stars less of a distraction because most posts don't have extras.