Most reliable metric: ignore 5 star reviews and only compare the number of 4 star reviews to the number of 1 star reviews. this works across all review sites. most sellers only fake 5 star reviews. however, if you review products and services that you know are quite excellent, you will that they often have way more 4 star reviews than one star reviews and more 4 than 3, more three than 2, and more 2 than one. this is because they are consistently excellent. but products with polar review metrics like all 5 star reviews and all 1 star reviews generally i find have the most fraudulent reviews and are most likely to have a bad experience. I think you should weigh 1 star reviews much higher than 5 star reviews. the new amazon metric or the advertisements talking about their aggregate quantity of five star reviews are wrong and misguided.
1 star reviews can also be paid for. I tend to look at 1/2/3-star reviews that seem passionate (was written like a real person with actual emotions - bonus points for photos) and see if the things that bothered them would also bother me.
I like your idea though, and will use it as an extra data point.
I tend to ignore the 1 star reviews because, in my experience, most are either from disgruntled people who wouldn't even try to open the manual first, or absolute binary thinkers.
When it comes to more subjective categories, like books, 1 star reviews are even worse.
1 star reviews are often skewed by the "delivery company did something dumb" reviews or a part was missing or there was a problem with returning it or that kind of thing. I often just look that the average rating isn't complete trash, if it's averaging 2 star on a lot of ratings the product probably is not worth considering. Then I pick some longer 4-star reviews and read the text. In the long reviews someone's put some thought into it and they're often more balanced.
User contributed ratings have, it seems, mostly outlived their usefulness. It isn't that things can't be reviewed, but ratings are like anything else in life, in that they have deaths. You want me to trust ratings more, let me order them by date.
Oddly, Steam does a good job with this. Two ratings, recent and all time. Google, amusingly, only shows the all time on the search page, but the recent reviews are very relevant.
Does this have to apply to all things? Maybe not, but manufacturing changes. needs of the time change. General expectations change. A 4 star rating from a decade is of much less reliability than one today. Even if it is critically a good rating.
I think this highly depends on what is being rated. Non-durable consumer goods, sure, there is very little assurance that what you're getting now is even what you would have gotten ten years ago. Processes and inputs both change. A previously reputable brand might have been purchased by private equity and enshittified on purpose.
On the other hand, if you're reading reviews of Moby Dick, I don't think the perceived quality or relevance of the writing is going to change much on the scale of decades. Maybe it starts to get archaic at some point, but people still get a lot out of reading Homer. If it's something like Principia Mathematica, that might honestly be timeless as long as human-like creatures exist who can read and understand math and physics.
I've found that sorting Amazon reviews by date works pretty well. It isn't uncommon to see all 5 star reviews for a while and mostly 1 star reviews in the more recent history.
My system is to just focus on the 1 & 2 star ratings, and look for trends, specifically. It takes a bit of work, since most of those reviews are just nonsense, but eventually true flaws in the product will be exposed through repetition. Due to the amount of work, I typically only reserve this method for big ticket items that would be difficult to return, such as appliances.
> You want me to trust ratings more, let me order them by date.
Interesting idea.
Having been involved with multiple third party sellers on Amazon, my extension of your thought is to have the ability to display a graph of all five rating levels over time.
This makes rating manipulation visible at both ends of the scale.
I know of people who, for example, sent out free samples of their product to a thousand people or more with a suggestion to post a four or five star review (after which they would get more goodies for free).
I also know of sellers who were attacked by either bots or a mob (in one case a competitor hired a team outside the US) who post hundreds of 1-star ratings and negative reviews.
If you could see a ratings graph over time, you should see step changes. This would be an indication of rating manipulation, both on the positive and negative side.
Many years ago I helped the seller who was the target of the fraudulent 1-star scheme prove to Amazon they were being attacked. The attack also consisted of said mob clicking on this seller's ads very early in the morning to exhaust the daily budget. So, bad reviews and your ads don't show-up when people are shopping. They got destroyed within a couple of months.
Graphing a bunch of this activity clearly showed rate-of-change discrepancies. It took about six months for Amazon to finally admit this happened. They had to refund over $100K in advertising that had been consumed through fraudulent clicks. Amazon would no reveal who did this (they claimed they knew, yet refused to provide the information to enable legal action). Sadly, they did not remove any of the fraudulent ratings and reviews.
Every single one of this seller's products absolutely tanked in search, going from pages 1 ~ 3 to 15+. Faced with the reality of the "mafia" behind the scenes when selling on Amazon, they felt they had no choice but to kill the business and focus on areas where open criminal manipulation and attacks were not tacitly allowed and ignored by the very ecommerce platform your business depends on. I know a bunch of people who got really hurt by Amazon through similar circumstances. To a person, most of them were people --not big business-- trying to make a go of selling product online to support their families and aim for a better life. Instead Amazon created an ecosystem that turned into a nightmare and caused most of them to lose money. Some even took out second mortgages on their homes to finance their entrepreneurial venture. Very sad.
Steam does that well, but its recommended/not recommended system is a dark pattern IMO. People tend to choose recommended even for games they don't like on the basis that someone else might like it, or it might improve in the future. Virtually every popular game has 'Overwhelmingly Positive' or similar. To get useful data you need to read the not recommended reviews in detail.
A more honest system would be 'Bad', 'Average' and 'Fantastic' - then I bet you'd get a ton more 'average' reviews.
At this point, aren't sellers gaming the system enough that the overall rating has become effectively meaningless? Individual reviews, maybe useful. But its not like the olden days where you could scour amazon for good products, ideally you would have a good idea what you want before even visiting the site.
I still find them useful. I use rating volume + ensuring rating is above 4 + checking whether the brand has a legit amazon store/landing page mostly to try and avoid duds. Individual reviews are useful for specific questions. As much as it sucks these days it's still better than going to your local durable goods retailer, having only one option to choose from with no ratings at all.
I do my research elsewhere (which has it's own problems), and then check Amazon for specific products (and these days, they often aren't any less expensive for name-brand items).
It doesn't look overcomplicated. When the average goes up, the number of 5 stars ratings (usually) goes up too, but the distinction between a 4.4 and a 4.6 is harder to spot than that between 40% and 60% 5 star rating (in this example, all ratings are either 4 or 5). So you could argue it offers a better discrimination.
But, it also really looks like they want to hide information that would stop you from buying a product. I know the "don't attribute to malice" maxim, but this is Amazon: they know what they're doing. You don't get to be this size by just fumbling around. They are obsessed with the look of their pages, keeping changes minimal, so they've thought about this long and hard. This change does look like a dark pattern.
> From a user experience standpoint, the new rating display system isn’t as glanceable as the five yellow stars. Additionally, the mention of “5-star” and the use of just one yellow star in the new arrangement could mislead new users, at least briefly.
It's overcomplicated enough to have 2 different star indicators.
It's an idea that's a few decades old. The late 90s had Bizrate.com managing millions of star images (via java apps) for retail ratings.
Amazon has always been the leading company in the word for rapidly shipping and testing changes. Fumbling around incrementally is their strategy. They made A/B Testing and Multi-Armed Bandit household terms.
Well there is always increased returns and a drop in overall purchases because you believe the rating is bullshit. Maybe they were already so bad on both, that this lifts overall sales long term.
I was so confused by this yesterday. I want to see the number of ratings compared to the average rating. That's how I know if a large number of people have tested this product and reviewed it properly. This system seems designed to hide that. You have to open every product to see that info.
It reminded me how important information design is.
Amazon has an average number of stars there, but it's no longer visually shown - instead, just a number. I have relied on the stars being "filled up" for so long that it now felt like every product was basically the same: a single, lone star. It was so harder to get an at-a-glance feeling of which products were good and which not-so-good.
Amazon is nothing but a giant flea market these days.
Everything is 5 stars through fake reviews and everything lies about it's capabilities.
Just today I was looking for a hairdryer and the top one advertised an RPM of 150,000. That'd put its 2 inch turbine spinning at almost 900mph. How much you want to bet the sound barrier isn't broken every time you turn that hairdryer on?
Honestly I've mostly stopped shopping online entirely. B&M stores have limited space, which means they have to choose what to stock, so they do the work of selecting decent products for me. When I do shop online, I don't use any of the flea markets (Amazon, Newegg, Walmart, etc).
I paid extra to buy drawer slides yesterday from another retailer for the same reason. One of the top sellers on Amazon were some drawer slides advertising as soft close when of course they were not. When the title is misleading, what the hell else is?
I'll pay more to not have to pick through the junk. And I say that as a person who loves to treasure hunt, but I don't want to spend hours reading reviews to find a decent product.
It baffles me that Amazon doesn't use their shopper's data more effectively. They could have gone with a weighted average, accounting for number of reviews, consistency of scores, etc. of the shopper in question.
Instead, they want to highlight the most deceptive side of the scores, i.e. 5-star ones.
Kinda feels like they don't want to end fake reviews, but prop them.
I should invent a statistical distribution for scores in 5 star rating systems. There should be peaks at 5 and 1 with a spread in between. Deviations from this distribution would be a fraud indicator, and the 5:1 ratio the rating.
With moves like this Amazon are trending towards the same discoverability quality in their marketplace that Google has achieved with the web as a whole, which may hint at another unknown fundamental law of internet economics.
i agree. i researched reviews for products and services i like and product and services that look fake. i think the top restaurants in my area have a staircase like distribution where 1 star reviews have the lowest aggregate quantity. products and services that suck have more one star reviews than 2, 3 and 4 star reviews and are consistently more polarized with one and five star reviews.
Normalise each person's reviews to make 3/5 their average.
Take notice when people do not leave a review who normally do (hello AirBNB! this is the metric you want!), it often indicates something has gone quite wrong.
I mean five star ratings have always been flawed. Any system where getting 4/5 is bad means the raters are confused, and then it's GIGO. This weird % five-star thing is a workaround for that GIGO because they're treating 4 stars as the same as 1 star.
Personally I'd switch to a "good/flawed/terrible" rating system and just give viewers a pie-chart (or a stacked-bar) of ratings.
I find the indicator for how often an item has been purchased in the last month (on the European site) far more helpful. On a side note, galaxus.ch even shows you the return rate.
[+] [-] daft_pink|2 years ago|reply
[+] [-] WalterSear|2 years ago|reply
The practice of switching products after garnering high review numbers is so widespread, Amazon can't pretend it's not tacitly complicit.
[+] [-] rjh29|2 years ago|reply
I like your idea though, and will use it as an extra data point.
[+] [-] manuelabeledo|2 years ago|reply
When it comes to more subjective categories, like books, 1 star reviews are even worse.
[+] [-] IMSAI8080|2 years ago|reply
[+] [-] Cthulhu_|2 years ago|reply
Uber's app might be perfect, but the rating will reflect on the company, drivers, and experiences of users.
Anecdotal, I've worked on apps for a train operator and currently a power company, it still holds up.
[+] [-] gunapologist99|2 years ago|reply
[+] [-] taeric|2 years ago|reply
Oddly, Steam does a good job with this. Two ratings, recent and all time. Google, amusingly, only shows the all time on the search page, but the recent reviews are very relevant.
Does this have to apply to all things? Maybe not, but manufacturing changes. needs of the time change. General expectations change. A 4 star rating from a decade is of much less reliability than one today. Even if it is critically a good rating.
[+] [-] nonameiguess|2 years ago|reply
On the other hand, if you're reading reviews of Moby Dick, I don't think the perceived quality or relevance of the writing is going to change much on the scale of decades. Maybe it starts to get archaic at some point, but people still get a lot out of reading Homer. If it's something like Principia Mathematica, that might honestly be timeless as long as human-like creatures exist who can read and understand math and physics.
[+] [-] _ea1k|2 years ago|reply
[+] [-] stronglikedan|2 years ago|reply
[+] [-] robomartin|2 years ago|reply
Interesting idea.
Having been involved with multiple third party sellers on Amazon, my extension of your thought is to have the ability to display a graph of all five rating levels over time.
This makes rating manipulation visible at both ends of the scale.
I know of people who, for example, sent out free samples of their product to a thousand people or more with a suggestion to post a four or five star review (after which they would get more goodies for free).
I also know of sellers who were attacked by either bots or a mob (in one case a competitor hired a team outside the US) who post hundreds of 1-star ratings and negative reviews.
If you could see a ratings graph over time, you should see step changes. This would be an indication of rating manipulation, both on the positive and negative side.
Many years ago I helped the seller who was the target of the fraudulent 1-star scheme prove to Amazon they were being attacked. The attack also consisted of said mob clicking on this seller's ads very early in the morning to exhaust the daily budget. So, bad reviews and your ads don't show-up when people are shopping. They got destroyed within a couple of months.
Graphing a bunch of this activity clearly showed rate-of-change discrepancies. It took about six months for Amazon to finally admit this happened. They had to refund over $100K in advertising that had been consumed through fraudulent clicks. Amazon would no reveal who did this (they claimed they knew, yet refused to provide the information to enable legal action). Sadly, they did not remove any of the fraudulent ratings and reviews.
Every single one of this seller's products absolutely tanked in search, going from pages 1 ~ 3 to 15+. Faced with the reality of the "mafia" behind the scenes when selling on Amazon, they felt they had no choice but to kill the business and focus on areas where open criminal manipulation and attacks were not tacitly allowed and ignored by the very ecommerce platform your business depends on. I know a bunch of people who got really hurt by Amazon through similar circumstances. To a person, most of them were people --not big business-- trying to make a go of selling product online to support their families and aim for a better life. Instead Amazon created an ecosystem that turned into a nightmare and caused most of them to lose money. Some even took out second mortgages on their homes to finance their entrepreneurial venture. Very sad.
[+] [-] unknown|2 years ago|reply
[deleted]
[+] [-] rjh29|2 years ago|reply
A more honest system would be 'Bad', 'Average' and 'Fantastic' - then I bet you'd get a ton more 'average' reviews.
[+] [-] unknown|2 years ago|reply
[deleted]
[+] [-] juujian|2 years ago|reply
[+] [-] fullshark|2 years ago|reply
[+] [-] alistairSH|2 years ago|reply
I do my research elsewhere (which has it's own problems), and then check Amazon for specific products (and these days, they often aren't any less expensive for name-brand items).
[+] [-] unknown|2 years ago|reply
[deleted]
[+] [-] tgv|2 years ago|reply
But, it also really looks like they want to hide information that would stop you from buying a product. I know the "don't attribute to malice" maxim, but this is Amazon: they know what they're doing. You don't get to be this size by just fumbling around. They are obsessed with the look of their pages, keeping changes minimal, so they've thought about this long and hard. This change does look like a dark pattern.
[+] [-] Supermancho|2 years ago|reply
It's overcomplicated enough to have 2 different star indicators. It's an idea that's a few decades old. The late 90s had Bizrate.com managing millions of star images (via java apps) for retail ratings.
[+] [-] hgsgm|2 years ago|reply
[+] [-] AJ007|2 years ago|reply
[+] [-] jasonjamerson|2 years ago|reply
[+] [-] karmelapple|2 years ago|reply
Amazon has an average number of stars there, but it's no longer visually shown - instead, just a number. I have relied on the stars being "filled up" for so long that it now felt like every product was basically the same: a single, lone star. It was so harder to get an at-a-glance feeling of which products were good and which not-so-good.
And unforutnately, that's probably Amazon's goal.
[+] [-] malfist|2 years ago|reply
Everything is 5 stars through fake reviews and everything lies about it's capabilities.
Just today I was looking for a hairdryer and the top one advertised an RPM of 150,000. That'd put its 2 inch turbine spinning at almost 900mph. How much you want to bet the sound barrier isn't broken every time you turn that hairdryer on?
[+] [-] coldpie|2 years ago|reply
[+] [-] knappe|2 years ago|reply
I'll pay more to not have to pick through the junk. And I say that as a person who loves to treasure hunt, but I don't want to spend hours reading reviews to find a decent product.
[+] [-] martin_a|2 years ago|reply
[+] [-] Pxtl|2 years ago|reply
[+] [-] manuelabeledo|2 years ago|reply
Instead, they want to highlight the most deceptive side of the scores, i.e. 5-star ones.
Kinda feels like they don't want to end fake reviews, but prop them.
[+] [-] fidotron|2 years ago|reply
With moves like this Amazon are trending towards the same discoverability quality in their marketplace that Google has achieved with the web as a whole, which may hint at another unknown fundamental law of internet economics.
[+] [-] daft_pink|2 years ago|reply
[+] [-] noja|2 years ago|reply
Take notice when people do not leave a review who normally do (hello AirBNB! this is the metric you want!), it often indicates something has gone quite wrong.
[+] [-] Pxtl|2 years ago|reply
Personally I'd switch to a "good/flawed/terrible" rating system and just give viewers a pie-chart (or a stacked-bar) of ratings.
[+] [-] trickpattyyo|2 years ago|reply
https://gist.github.com/trickpattyFH20/8d02b18aaf63b7d405456...
[+] [-] soneca|2 years ago|reply
I don’t have an opinion if this is good or bad for customers yet.
[+] [-] marban|2 years ago|reply
[+] [-] tguvot|2 years ago|reply
[+] [-] mvuijlst|2 years ago|reply
[+] [-] hgsgm|2 years ago|reply
[+] [-] ericjack427|2 years ago|reply
[deleted]
[+] [-] hulamedulla|2 years ago|reply
[deleted]
[+] [-] unknown|2 years ago|reply
[deleted]