About
Reading
Writing

The problem with crowdsourced reviews

2022 February

The problem with crowdsourced reviews is two-fold. People refuse to admit that most experiences that they have are average (by definition), and additionally refuse to use the entire range of ratings at their disposal.

Dunkey described the problem eloquently and comically in Game Critics, but his criticisms are valid for every kind of crowdsourced reviews.

Let's assume that we're rating restaurants on a five-star scale. Most people think that the average restaurant they go to deserves three or four stars. A good restaurant? Four stars. Slightly worse than average? Three stars.

Let's formalize this method of rating restaurants:

  • 1/5: Unused
  • 2/5: Unused
  • 3/5: Worse than Average, Average, Good.
  • 4/5: Average, Good, Great.
  • 5/5: Great, Amazing.

The issue with the above system is that even with a large sample size, nothing useful can be gleaned. Not only that, the lack of one and two-star reviews imply that no restaurants are bad! It's astounding that people rarely complain about food they've spent money on, but this problem exists in nearly every other category as well.

Here's a much more sane way of rating restaurants:

  • 1/5: Below average. A small minority of restaurants fall in this category.
  • 2/5: Average. The majority of restaurants fall in this category.
  • 3/5: Better than average, worth another visit.
  • 4/5: Great. Worth recommmending to others.
  • 5/5: Amazing. I'd recommend this to everyone I know.

This system of reviewing uses all five ratings, notably accepting the truth that the majority of restaurants that we go to are average, which is true by definition.

Why is this better? Currently, when I check the reviews for a restaurant in NYC, I always see a number between 4.0-4.4. It's gotten to the point where my brain doesn't even register the rating, as it carries absolutely no meaning despite the thousands of people who have "contributed".

If we all used the rating system described above, it would be possible for our collective crowdsourced opinion to matter.

Note: Restaurants are just an example. This applies to any crowdsourced vote with a non-binary voting option.