I fully understand the frustration the authors may feel when a troll with nothing to prove their expertise starts giving bad ratings or irrelevant advice.
I have pulled some data from the gallery database and currently there are 5504 reviews. 4349 of them were written by people, who have also uploaded at least 1 icon or cursor set, 1155 were done by people, who never uploaded anything.
Overall, the number of stars given to a set does not seem to differ too much between authors and lurkers. Authors have given 41% of 5-star rating and 8.8% of half-star ratings. Non-authors are actually a bit more generous - they have given 48% of 5-star ratings and 8.2% of half-star ratings.
The final rating of a set is computed as a simple average of the individual ratings. This probably is not the best way, some ratings are obviously more representative and should be given more weight. Though figuring out the right formula may be tricky.
An alternative would be drop the star rating system and replace it with a simple Like button or maybe with Like/Dislike options. Then the number of Likes divided by a number of page views would be representative of the set quality and could be expressed by stars. But I do not have the page view statistics on hand - I have added a counter week or two ago, but it is unreliable - you can see the number of page views when you hover your mouse over the download counter.