It's a bit harder to separate reviews between countries on Android, but we have 7,862,505 recent reviews indexed in the last 2 months on Android, and 780,997 of them are negative (9.93%). I will try to run a query with language detection, though that might take a bit of time~
Thank you, this is cool. It's very annoying to read comments like "I gave 1 star but it's a good game" however I wonder how much is the actual effect on the ratings.
Maybe it's a Saturday night thing but the discussion for this part mostly has comments that rival the ignorance of the OP, e.g. "The sad fact is that most people are awful", "you could block that country from your app", all based on simplistic anecdotal analysis given in the post.
Before you arrive at such a sweeping conclusion, you should perform at least a little bit of statistical testing, e.g. see diziet's analysis on this page.
Two further comments:
1. The behavior mentioned in the post does seem to be a real thing, as evidenced by the many entries in the Turkish zeitgeist site eksisozluk: https://eksisozluk.com/yorumum-gozuksun-diye-1-yildiz--41109..., the first entry under this heading is dated 11/13/2013 so it seems to be a new meme. Also, the entries suggest that the behavior seems to be focused on Google Play rather than the Apple Store.
2. This shows a significant drawback of Google Play's selection of comments to display. Looking at the selected comments for the same game on the Apple Store (https://itunes.apple.com/us/app/wheres-my-water-2/id63885314...) I don't see this kind of 1 star ratings.
You wouldn't see these ratings on the Apple Store unless you are in Turkey (or changed your device's region) because Apple separates app store ratings. US devices only see US ratings. Turkey only sees Turkey ratings.
EDIT: Also, Apple doesn't display ratings like Google does, so there is clearly no reason for them to leave reviews like that.
Hi, I am aware that the post on my blog is pretty shallow but my intention was not to dive deeper into the problem anyway.
As you confirmed, the phenomenon is real. I am an IOS user and I was shocked when a friend told me about the comments people are making at Google Play, so I quickly put together a blog and wrote a post that dramatically exposes the problem. The idea is to catch attention of the people so that those who have the necessary tools and time can investigate it thoroughly.
The problem is real and hopefully somebody in the position to fix it would see the post.
If you are going to take user reviews seriously, you have to account for the possibility that a significant portion of the general population may be comprised of idiots.
People misusing the star system is a problem, but the real problem is that we are forced to care when people misuse it.
(To correct this specific problem, I would recommend low-tolerance hellbanning. If a positive review is accompanied by a one-star rating, that user's ratings and comments will never be seen by another user again.)
Are developers on the play store allowed to delete reviews? That sounds like something that would be really easily abused (i.e. delete all bad reviews of your bad product).
I am Turkish, I live in the US and run a startup. I conducted a small experiment (n=3) among my cousins living in Turkey aged 8-14: They don't understand that 3rd parties make these apps not tied to the device; and they definitely don't understand that the rating (in general) would impact someone making this app financially. Huh.
I don't think this phenomenon has much to do with user misconceptions, it's more of a meme that has spread to many Turkish users, e.g. do a search on "gozuksun diye 1 yildiz verdim" (gave 1 star so my comment is seen) or the eksisozluk link in my comment below.
Why do you think this is more prevalent in Turkey than in other countries? Is there something unique about Turkish culture that would cause this misunderstanding or misprioritization?
So you say that you asked a bunch of questions to a couple kids and concluded that the whole population of Turkey are idiots? Let me tell you, your cousin's reactions are quite expected, but I doubt yours are.
Turkish users want their reviews to be seen. This sounds narcissistic to us, but for a lot of people being asked for their opinion of a product is a big deal. Rather than condemning the users, we should look for ways to improve the UX of the Google Play Store.
There could be cultural factors at play here as well. When I was growing up in Turkey, every so often there would be a fad that picked obscure artefacts of American culture and turned them into articles of social prestige. Cheesy American soap operas, songs and actors would suddenly and inexplicably become national sensations that everyone had to know. I assume something similar is going on with the Android Play Store reviews. Having your review on the frontpage of an app had become a point of pride, so much so that users are subverting the system to get there.
It's kind of weird that they want their comment to be at the top, but the only thing they say in their comment is that they want it to be at the top. You'd think if they were looking for exposure, they'd use the exposure to do something other than talking about how they wanted exposure.
This makes me wonder... are there locales where 1 star might be taken to mean a better review than a 5 star review?
I realize that's not what's being claimed here, though.
There's no intrinsic meaning to how stars correlate to quality is my thinking here, but perhaps only due to familiarity do we think that more stars is better and not worse.
It seems at least in this case they know what they are doing because they say thing like this "Great game, but give it 1 start so people see my comment". I think they are fully aware and are even "apologizing" for it.
Here's a comment which suggests that at least some of them are thinking 1 star is the best:
Kubilay Tekin: I hadn't gave 1 star just because I want my comment gets shown on top, I actually think that the game is aweful aqws.w.wew..ws
(I've seen this phenomenon before - there are quite a few who associate 1 with first place, hence surveys and such often have the [1 = worst, 10 = best] next to each "rate xxx on a scale of 1 to 10")
Indeed there are many rating systems where '1' is 'best', 'first-in-rank', 'most-preferred'.
I've never seen an online 'stars' system that works that way... but it's possible in some other cultural context a single star is somehow honored more than many, and thus without explanation the star-system could be misinterpreted. (Of course the comments here strongly suggest that's not a factor in this case.)
Maybe star ratings should be removed in favor of text reviews with "ratings" based exclusively on sentiment analysis. Has machine learning reached a state where that's practical? If not, will anyone hazard a guess on when it will?
WTF? Why is this bug report getting spammed with talk about Turkey? The bug is that the system displays one star reviews first, creating broken incentives. Assuming that the code works (i.e., doesn't work) the same way in every country, the country has nothing to do with it. The system needs to be fixed to use a more neutral display order.
Is there anyone else who thinks the "5 star" rating is somewhat flawed, or is it just me?
I think, the problem is that the scale is all on the positive side, but people are trying to use it to give negative rating. So naturally they readjust the scale, so that it has both, positive and negative sides, by "shifting" the zero of the scale. Now, where exactly does it go? Will 3 become 0, or is it 2 for some people, or maybe 4?
I know it's way too late in the game, but why can't the rating system be just a simple "good", "bad", "it's ok", or maybe even: :-(, :-|, :-)
I am not enamoured with the "5 star" system, but I have to say that I like it more than the standard 5-choice system that explicitly labels the choices as positive or negative: Strongly Disagree, Disagree, No Opinion, Agree, Strongly Agree
I despise that sort of system. None of the choices ever seem to be appropriate for what I think or feel.
Star systems avoid inspiring this frustration in me by giving me the freedom to imagine labels that I think fit how I feel or think.
When Netflix had their contest to improve recommendations, one of the common first-pass techniques on the forums was to zero-adjust the ratings for each user. It's hard to imagine that a big-data company of Google's size couldn't do this already, although who knows if they are.
Croatian school system uses grades 1 to 5. 1 is fail, 2 is pass, 3 is good, 4 is very good and 5 is excellent. Teachers sometime also add a minus or plus in front of the grade, e.g. -5 is almost excellent and 4+ is a bit better than very good.
Is it for mixed reviews as in total reviews worldwide and Turkey or just for Turkey. It seems it is just for Turkey.
Maybe they are just more honest and say why they are giving 1 star reviews. Or maybe they are dishonest and really think the game is terrible but in the comment chose to say something else to seem nice? Is there somehow a disproportionate presence of Turkish Android app developers and since they are competing with you they are just messing with author's head.
Looking at the game review in the app store (presumably the US version) most 1 or 2 star reviews seem valid (and don't see a particular trend with names, to mean they are certain ethnicity).
A guess it would be good to browser other apps' reviews from Turkey. So I took a look at Cut The Rope. Very popular game, indeed.
Hmm, well I can't read Turkish. So I opened Google Translate and started translating a few top 1 star review. And yap, same pattern.
===
Mükemmel Çok güzel harika bir oyun gözüksün diye 1
---
Excellent Very nice people to see a great game as one
===
===
Gerçekten cok guzel bir oyun hem zeka açıcı Gercekten cok güzel
---
Really a very nice game're really nice and intelligence opener
===
Someone might want to help, but even with Google Translate it looks like they are giving it good reviews as text but 1 star as a score.
The crude and seemingly insensitive way is to just prevent Turkish reviewers from commenting. Or even better hellban them and never account for their score. The would correct this pretty quickly I would imagine.
Now I would really hope someone from Turkey to explain if there is a cultural or social reason for this. It just seems to strange and odd.
This is really odd, particularly considering that Turkey is by far one of the friendliest places I've been in my many travels (well, at least many travels when I was younger).
I question whether they're aware of the grief that their reviews are causing. In any case, it's still pretty inconsiderate to tell someone (numerically) that you think their work sucks, even if you don't really mean any harm.
All in all, probably an extension of the whole "cursing/attacking/belittling someone on the Internet doesn't count" phenomenon.
I hate the iOS review system. People rarely leave a review if the like the app, but as soon as they have something to complain about they will leave a bad review. Also most people if they have an issue will automatically leave a bad review instead of contacting the support email to get it resolved.
What's even more annoying is people that write things like 'Best app ever' and then give it a 3 or 4 star rating instead of 5.
> Also most people if they have an issue will automatically leave a bad review instead of contacting the support email to get it resolved.
because most people are used to working with big products. Even me, a developer, this is the first time I considered that you could in general email support and not get some patronizing useless robot. It makes sense now that I think about it, since so many of the apps are made by small sized development firms or even just lone wolfs, there's a good chance I could get a human, maybe even another dev!
In general, that would never happen with a large and well known product. Faced with that, people take the only outlet they feel with maybe change something.
I think you'll see the same pattern all over the Internet, whether it's app stores or forums. People have to go out of their way to post something, and human nature is such that they're much more likely to expend that effort to post a negative opinion on something than a positive opinion.
I wonder if at least some of the motivation for "Please rate us!" popups in mobile apps isn't just to get more ratings, but also to get a more representative (i.e., more positive) sampling of user opinions as well.
I barely ever see reviews and have to ask to see reviews for other version. I'm in New Zealand. Its a disincentive to updating an app as far as I can tell. If something works and isn't broken, minor updates will lose the developer prominent reviews. Or am I missing something?
Man, with PC games you have forums so that's great for communicating, but with mobile it seems like there's generally a disconnect between developer and player. The system, while great to have, doesn't really encourage it.
We had the same problem, but with Russia. Ours being a paid app and our reviews being in the low 1000s, every 1-star rating was hurting us badly, so we removed Russia completely from our distribution. We added it back a month later and things went back to normal, but that's definitely something that we shouldn't need to have done.
Maybe Google should do sentiment analysis on review text and compare with star rating then rank the review to be less visible if the two contradicts too much?
As a Turkish, I think i know little better what is going on here;
It is the high school kids that they think it is cool to mess up with big and serious systems in their stupid way. It is fun for them to mess up rules which I think is the same for most of the teenagers around the world. They try to break things, don't listen teachers, don't go to school.
They think that they should be doing what they want instead of what others want them to do.
The outcome for the turkish teenagers irritating behaviour showed in Google's Play Store. That is all what i can see.
Aren't we all tired of these kids actions around us sometimes? :)
People's review habits in general are bizarre. I have read reviews like "I haven't gotten a chance to use the product yet" with a 3 star rating. One wonders what exactly they are reviewing. Or recipe reviews are the weirdest, every one of them say "I change blah blah blah" to the point where its a completely different recipe then they review their recipe, like that's somehow appropriate. I can understand making one tweek to suit your taste but don't change the whole recipe and think thats what you are reviewing.
Mere speculation: Isn't this just a result of the mixing of "Helpfulness" feature and the black-hat marketing / reviews purchasing?
As in now those selling "Reviews" for your app; something like $100 for around fifty 3 to 5 star reviews can justify their price by stating their reviewers have consistently ranked as being "Helpful" and always on the "front page" of any app review? And what we're seeing is the reviewers working for these black-hat intermediaries trying to up their overall review "weight"; whereby I'm assuming google takes into account your previous reviews (and their helpfulness) with how long your review now stays on the front page (much like comments staying on top of a HN thread).
Or it could just be a typical fad akin to wanting the personalized car registration of "A1", being top/first etc.
[+] [-] diziet|12 years ago|reply
On iOS, we have 32,296,341 reviews, with 3,448,351 being 1 star reviews. 10.67% one star reviews.
Let's see if we limit it to reviews in the last 2 months:
16.47% negative in Turkey (65,154 total reviews) 12.05% negative in the US (1,960,283 total reviews)
(source - internal https://sensortower.com analytics)
It's a bit harder to separate reviews between countries on Android, but we have 7,862,505 recent reviews indexed in the last 2 months on Android, and 780,997 of them are negative (9.93%). I will try to run a query with language detection, though that might take a bit of time~
[+] [-] smilliken|12 years ago|reply
There's definitely a bump in 1-star reviews in Turkey.
[+] [-] mrtksn|12 years ago|reply
[+] [-] userbinator|12 years ago|reply
For a second I misread that as 65,536 (64K) reviews.
[+] [-] Jun8|12 years ago|reply
Before you arrive at such a sweeping conclusion, you should perform at least a little bit of statistical testing, e.g. see diziet's analysis on this page.
Two further comments:
1. The behavior mentioned in the post does seem to be a real thing, as evidenced by the many entries in the Turkish zeitgeist site eksisozluk: https://eksisozluk.com/yorumum-gozuksun-diye-1-yildiz--41109..., the first entry under this heading is dated 11/13/2013 so it seems to be a new meme. Also, the entries suggest that the behavior seems to be focused on Google Play rather than the Apple Store.
2. This shows a significant drawback of Google Play's selection of comments to display. Looking at the selected comments for the same game on the Apple Store (https://itunes.apple.com/us/app/wheres-my-water-2/id63885314...) I don't see this kind of 1 star ratings.
[+] [-] clarky07|12 years ago|reply
EDIT: Also, Apple doesn't display ratings like Google does, so there is clearly no reason for them to leave reviews like that.
[+] [-] mrtksn|12 years ago|reply
As you confirmed, the phenomenon is real. I am an IOS user and I was shocked when a friend told me about the comments people are making at Google Play, so I quickly put together a blog and wrote a post that dramatically exposes the problem. The idea is to catch attention of the people so that those who have the necessary tools and time can investigate it thoroughly.
The problem is real and hopefully somebody in the position to fix it would see the post.
[+] [-] diziet|12 years ago|reply
Compared to the US, which received a lot of negative reviews (https://sensortower.com/ios/us/disney/app/wheres-my-water-2/...) and has a larger sample size. It could be a much more recent behavior.
[+] [-] Crito|12 years ago|reply
People misusing the star system is a problem, but the real problem is that we are forced to care when people misuse it.
(To correct this specific problem, I would recommend low-tolerance hellbanning. If a positive review is accompanied by a one-star rating, that user's ratings and comments will never be seen by another user again.)
[+] [-] robertjwebb|12 years ago|reply
[+] [-] dkural|12 years ago|reply
[+] [-] Jun8|12 years ago|reply
[+] [-] icambron|12 years ago|reply
[+] [-] jonny_eh|12 years ago|reply
[+] [-] gkya|12 years ago|reply
http://en.wikipedia.org/wiki/Scientific_method
[+] [-] picardo|12 years ago|reply
There could be cultural factors at play here as well. When I was growing up in Turkey, every so often there would be a fad that picked obscure artefacts of American culture and turned them into articles of social prestige. Cheesy American soap operas, songs and actors would suddenly and inexplicably become national sensations that everyone had to know. I assume something similar is going on with the Android Play Store reviews. Having your review on the frontpage of an app had become a point of pride, so much so that users are subverting the system to get there.
[+] [-] gfodor|12 years ago|reply
[+] [-] mrtksn|12 years ago|reply
[+] [-] auctiontheory|12 years ago|reply
[+] [-] baddox|12 years ago|reply
[+] [-] AJ007|12 years ago|reply
In a mass used platform a single user should never be able to receive 100% visibility by a simple action.
[+] [-] bagels|12 years ago|reply
I realize that's not what's being claimed here, though.
There's no intrinsic meaning to how stars correlate to quality is my thinking here, but perhaps only due to familiarity do we think that more stars is better and not worse.
[+] [-] rdtsc|12 years ago|reply
[+] [-] userbinator|12 years ago|reply
Kubilay Tekin: I hadn't gave 1 star just because I want my comment gets shown on top, I actually think that the game is aweful aqws.w.wew..ws
(I've seen this phenomenon before - there are quite a few who associate 1 with first place, hence surveys and such often have the [1 = worst, 10 = best] next to each "rate xxx on a scale of 1 to 10")
[+] [-] gojomo|12 years ago|reply
I've never seen an online 'stars' system that works that way... but it's possible in some other cultural context a single star is somehow honored more than many, and thus without explanation the star-system could be misinterpreted. (Of course the comments here strongly suggest that's not a factor in this case.)
[+] [-] pacala|12 years ago|reply
> Good, a very good game! I gave 1 star so that my comment gets seen.
++ very positive
> The game is an example of how a good game should be! Don't bother that I gave 1 star, it's because I want my comment gets seen.
++ very positive
> Super! I gave 1 star so that my comment is shown first but I actually liked the game.
0 neutral
> I hadn't gave 1 star just because I want my comment gets shown on top, I actually think that the game is aweful aqws.w.wew..ws
- negative
[+] [-] andrewflnr|12 years ago|reply
[+] [-] rwallace|12 years ago|reply
[+] [-] rytis|12 years ago|reply
I think, the problem is that the scale is all on the positive side, but people are trying to use it to give negative rating. So naturally they readjust the scale, so that it has both, positive and negative sides, by "shifting" the zero of the scale. Now, where exactly does it go? Will 3 become 0, or is it 2 for some people, or maybe 4?
I know it's way too late in the game, but why can't the rating system be just a simple "good", "bad", "it's ok", or maybe even: :-(, :-|, :-)
[+] [-] Crito|12 years ago|reply
I despise that sort of system. None of the choices ever seem to be appropriate for what I think or feel.
Star systems avoid inspiring this frustration in me by giving me the freedom to imagine labels that I think fit how I feel or think.
[+] [-] renata|12 years ago|reply
[+] [-] dsego|12 years ago|reply
[+] [-] rdtsc|12 years ago|reply
Is it for mixed reviews as in total reviews worldwide and Turkey or just for Turkey. It seems it is just for Turkey.
Maybe they are just more honest and say why they are giving 1 star reviews. Or maybe they are dishonest and really think the game is terrible but in the comment chose to say something else to seem nice? Is there somehow a disproportionate presence of Turkish Android app developers and since they are competing with you they are just messing with author's head.
Looking at the game review in the app store (presumably the US version) most 1 or 2 star reviews seem valid (and don't see a particular trend with names, to mean they are certain ethnicity).
A guess it would be good to browser other apps' reviews from Turkey. So I took a look at Cut The Rope. Very popular game, indeed.
Looking at US reviews, looks good.
Turkish:
https://play.google.com/store/apps/details?id=com.zeptolab.c...
Hmm, well I can't read Turkish. So I opened Google Translate and started translating a few top 1 star review. And yap, same pattern.
===
Mükemmel Çok güzel harika bir oyun gözüksün diye 1
---
Excellent Very nice people to see a great game as one
===
===
Gerçekten cok guzel bir oyun hem zeka açıcı Gercekten cok güzel
---
Really a very nice game're really nice and intelligence opener
===
Someone might want to help, but even with Google Translate it looks like they are giving it good reviews as text but 1 star as a score.
The crude and seemingly insensitive way is to just prevent Turkish reviewers from commenting. Or even better hellban them and never account for their score. The would correct this pretty quickly I would imagine.
Now I would really hope someone from Turkey to explain if there is a cultural or social reason for this. It just seems to strange and odd.
[+] [-] jnbiche|12 years ago|reply
I question whether they're aware of the grief that their reviews are causing. In any case, it's still pretty inconsiderate to tell someone (numerically) that you think their work sucks, even if you don't really mean any harm.
All in all, probably an extension of the whole "cursing/attacking/belittling someone on the Internet doesn't count" phenomenon.
[+] [-] sprite|12 years ago|reply
What's even more annoying is people that write things like 'Best app ever' and then give it a 3 or 4 star rating instead of 5.
[+] [-] GhotiFish|12 years ago|reply
because most people are used to working with big products. Even me, a developer, this is the first time I considered that you could in general email support and not get some patronizing useless robot. It makes sense now that I think about it, since so many of the apps are made by small sized development firms or even just lone wolfs, there's a good chance I could get a human, maybe even another dev!
In general, that would never happen with a large and well known product. Faced with that, people take the only outlet they feel with maybe change something.
The best app ever part. Well...
People always exaggerate, ALWAYS.
:)
[+] [-] gfodor|12 years ago|reply
Make sure you provide a clear way for them to contact you if they are running into trouble.
Reach out to your engaged users and pro-actively ask them to review the app after you're confident they have had a good experience.
[+] [-] TylerE|12 years ago|reply
[+] [-] mwfunk|12 years ago|reply
I wonder if at least some of the motivation for "Please rate us!" popups in mobile apps isn't just to get more ratings, but also to get a more representative (i.e., more positive) sampling of user opinions as well.
[+] [-] lostlogin|12 years ago|reply
[+] [-] cclogg|12 years ago|reply
Man, with PC games you have forums so that's great for communicating, but with mobile it seems like there's generally a disconnect between developer and player. The system, while great to have, doesn't really encourage it.
[+] [-] username223|12 years ago|reply
http://effyr.tumblr.com/
[+] [-] zrgiu_|12 years ago|reply
[+] [-] jnbiche|12 years ago|reply
[+] [-] xcyu|12 years ago|reply
[+] [-] greenwalls|12 years ago|reply
[+] [-] whizzkid|12 years ago|reply
It is the high school kids that they think it is cool to mess up with big and serious systems in their stupid way. It is fun for them to mess up rules which I think is the same for most of the teenagers around the world. They try to break things, don't listen teachers, don't go to school.
They think that they should be doing what they want instead of what others want them to do.
The outcome for the turkish teenagers irritating behaviour showed in Google's Play Store. That is all what i can see.
Aren't we all tired of these kids actions around us sometimes? :)
[+] [-] aestra|12 years ago|reply
[+] [-] _s|12 years ago|reply
As in now those selling "Reviews" for your app; something like $100 for around fifty 3 to 5 star reviews can justify their price by stating their reviewers have consistently ranked as being "Helpful" and always on the "front page" of any app review? And what we're seeing is the reviewers working for these black-hat intermediaries trying to up their overall review "weight"; whereby I'm assuming google takes into account your previous reviews (and their helpfulness) with how long your review now stays on the front page (much like comments staying on top of a HN thread).
Or it could just be a typical fad akin to wanting the personalized car registration of "A1", being top/first etc.
[+] [-] anigbrowl|12 years ago|reply