There is a thing in decision theory called the exploration/exploitation tradeoff. Exploitation is always choosing the option that is estimated to be best. Exploration is choosing other options to see if they might be even better. In this case showing new comments is exploration.
Hacker news has a nice solution to this problem. New comments appear at the top of the thread, and then slowly fall unless they get votes. I think this works very well, although I think it's still a bit too far on the exploitation tradeoff.
I've thought about how to solve this problem, and I think the best solution is votes/time since the comment was posted. New comments will immediately appear at the top, since they have infinite vote/time ratio. But they will quickly fall to their correct ranking.
There are other solutions, like treating it like a multi-armed bandit problem. And you can use something like https://en.wikipedia.org/wiki/Thompson_sampling . Create a probability distribution for the upvote/downvote ratio for every comment. Sample from that distribution, and rank the comments accordingly. This should be close to optimal, but it requires people to use the downvote button lots (because it measures comment quality by upvote/downvote ratio, which is what reddit does currently.) Without downvotes, you could use time again, and create a probability distribution for votes/time.
I would add that while it's plausible, it's by no means obvious that Reddit is leaving business value on the table. Imagine that they seek to minimize the bounce-rate on any thread such that they maximize overall time on site for each user. In this case, the important part could be that the top comment is 'not wrong' rather than 'right'. If the first thing users see is too factually incorrect, controversial, or emotionally upsetting, people could be driven people away from the site. Essentially this is the same mechanism that helps cause the Facebook news bubble effect -- people leave if you tell them what they don't want to hear, so you write an algorithm that underweights anything that might do so.
To put it into economic terms, if users have high loss aversion (from being upset, etc.) relative to their thirst for knowledge gained from great comments, you would expect exploitation to be overweighted vs exploration. The business value maximizing solution would then be that once an appropriate top comment is found, the thread should tend to stay in that equilibrium.
"I've thought about how to solve this problem, and I think the best solution is votes/time since the comment was posted. New comments will immediately appear at the top, since they have infinite vote/time ratio. But they will quickly fall to their correct ranking.
"
The traditional solution to this problem is wilson score intervals.
The real problem there is that most sites that use them only use one part of the bound, instead of using both.
Ideally, you use the lower bound to scale how good something is, and the upper bound to determine with what priority you show it order to get more votes on it to determine better bounds to be more confident of your lower bound.
This is what Google Moderator did, and pretty much never had these problems as a result.
(Is it perfect? No. You need a good place/way to show things for voting. But assuming you get that, it works very well in practice).
If you only use the lower bound, you get systems where you only ever become more confident in the scores of the things that show up first (like is happening here).
That said, it's still biased. Years ago, HN admins/mods marked my account as "low-quality content" so now my comments always quickly fall down to the bottom of the parent, no matter how many points they have. They admitted it in an email recently, and refused to take it off my account, despite the quality of my comments no longer being trollish. When you have human intervention like this, there's danger if it heavily biasing the entire forum for the worse.
This search for a better way to rank comments still, in my opinion, makes a large incorrect assumption; namely that it is desirable (and possible) to display comments in order from "best" to "worst". If you think of comments only from the prespective a viewer then yes ordering comments from best to worst makes sense, the viewer wants to get as much information as quickly as possible. But if you are looking at establishing a robust meaningful dicussion, part of that is that all comments should have a chance of getting responded to that is proportional to their value, not their rank. Realistically the top three comments in a large thread are about equally good, but the top comment is going the vast majority of views and replies. One system the facilities dicussion might effectivelly shard it's users into separate threads of managble size so they can have a real dicussion, though I'm fuzzy on the details.
Given all the worry we have about about how our online discussions work I think their is real societal (if not extrable monetary) value in discovering a better solution.
Back when I was a regular on LessWrong and it had an active community, it didn't have that problem at all, despite using a variant of the Reddit codebase. Their solution was pretty simple, just show the five most recent comments in the right hand column. People would interact with them, check out their parent comments, and threads would grow organically regardless of age. In fact people would often come back to very old threads and revive them. It might be tricky to scale that to larger communities though.
Yes, this non-linear idea seems very interesting. I have read several discussions on various boards with much hand-wringing about the best way of sorting comments - by time, by votes, by user rep, by some sort of weighted average of the above? Suddenly things get rather complicated as people try to decide how the weights should be balanced. But this idea of showing multiple different feeds side-by-side is very refreshing. Two (or even three?) feeds with different weightings and (and relative sizes!) could provide a better balance than what we are used to.
(As an aside, it would be entertaining to take it to an extreme - seeing the top voted comments also shows you the bottom voted, most controversial also shows least, etc. A sort of anti-echo-chamber measure.)
I like the simplicity of this solution, but it might be a good idea to combine it with other relatively simple measures that dampen the "first comment effect," such as randomizing which comments show up at the top, as others have suggested in other threads here.
Can you comment on how comments could benefit from NLP related advances? E.g. detecting entities and using those as entry points into discussion, detecting similar topics discussed in different threads, detecting comments which are low effort and don't add much value. I hate the fact that first/top comment sets the tone for discussion overall and lot of people end up expending energy on that topic alone. People end up discussing something tangential just because the first commenter had "interesting" observation. Also I hate the pun threads which plagues reddit.
Makes sense, due to a self-reinforcing dynamic: early comments that are good enough to gather upvotes will stay near the top of the page, gathering more upvotes.
I suspect HN is subject to the same "first comment effect," often to the detriment of later comments that might be more deserving of the top spot.
Have three categories: New, Best (up:down ratio is high), Controversial (up:down ratio of 1:1).
Pick one from each category and then randomly make them the top three. For larger threads you can salt them into the top 10 comments.
New comments that are truly good have a chance of being picked up to Best or Hot, and Controversial comments spur discussion and moderate the self-reinforcing circle-jerk that threads descend into.
Well, the trend is so smooth that it would be very simple to apply a modifier to comments to counter it. Effectively, votes would slowly expire. In order to remain relevant, a comment would have to be constantly upvoted.
I actually think this would be an interesting experiment to try. Who's hacker friendly and has a well-used comment system that we know?
I don't know if this would change the ranking but I've always wondered if it would be good if threads were ranked by the waited sum of all replies to it. In otherwords the original comment might not have a high vote but if a reply did, even a deep one, the thread might appear at the top. Or does it already work that way?
I am guessing that HN will have a lower percentage than reddit. New comments get a bit of time at the top and have a chance to gain votes if they are worthy.
This is surprising, and it could mean that early commenters have their voice heard more than later commenters. But it could also just mean that topics raised by early commenters end up being discussed in those threads, and commenters on both sides have their voices heard.
For example, if the first comment is roughly "I read the linked article and disagree because of A, B, and C" then someone with an opposing viewpoint would probably reply to the existing first comment instead of creating a separate comment.
This doesn't mean that the second commenter's voice isn't heard—it just means it all happens under the thread created by the first commenter.
If two commenters want to say the same thing, the first arrival will post it, and the second will upvote it (and maybe reply with additional information). So it's not surprising that top comments are weighted towards early arrivals. The question needs to be are there later comments that are objectively better that don't end up sorted higher when sorting by "best".
The "best" sort, by the way, does not use time as a factor, but rather generates a confidence score. Newer comments have less votes and thus less confidence (which ends up giving them a slight boost).
What if you consider it as an optimization problem for the late commenter? You have to choose where to insert your comment, most likely there are multiple options.
A root-level comment when there are already dozens is unlikely to be noticed; there is significant momentum behind the top comments, usually for good reason - whether jokes, clever insight, or a popular viewpoint.
Instead, if you reply near the top in a context where it makes some sense, you skip the root-level graveyard and are almost guaranteed good placement with the downside of "overhead" due to the upwards chain always appearing first.
Consider also that the penalty for being the 2nd, 3rd, or worse comment at root-level gradually increases due to the increasingly nested replies to each highly rated comment. If you assume root-level comments express opinions, minority opinions are also more likely to maintain good position in a reply comment such as the one you posited.
Lastly, "trickle down karma" could be a factor. A quality reply to a root-level comment could enhance the parent comment's value and result in additional karma for them. Such situations could include biases like assuming comments which merited replies are interesting due to peer validation and thus reading them, and more fundamental added value such as uncited sources being given.
I do disagree with one point: that there would be no correlation between time posted and upvotes if the reddit algorithm were completely fair. If I post a link to r/tipofmytongue asking "What was that low-budget movie that had engineers inventing a time machine in it?" and the first comment correctly tells me I was thinking of Primer, then it'll be highly upvoted. But later commenters won't get to grab that easy karma since it's already answered. There are only so many "great" things to say in response to most posts. Also, if someone posts in a thread after it has dropped off the front page, they'll get fewer upvotes regardless of what the algorithm does.
The reason is most likely that the top and bottom of the page are easiest to find so the comments at the bottom have a little more visibility than those in the middle.
The HN algorithm changed very recently, which was one of the reasons I had the idea to take a closer look at the Reddit data. HN, within the past few months, will highlight new comments within the first 2 slots even on busy threads.
I would perform the same analysis on HN data to confirm if I could. (HN does not expose comment scores)
Well, this is one reason to consider normalizing up/down votes by views. Is there any data on the number of views each comment received (number of times each was scrolled into view, for example)?
You'd also have to factor in the likelihood of each person up/down voting (if it's viewed by many people who weren't going to vote anyway, then no votes are nothing out of the ordinary).
The same thing happens with search engines as well, where items at the top of the search results tend to stay at the top because they're purchased or clicked on at a higher rate. Can anyone chime in with what they've done to fix this in a search context? Has anyone had success with a one-armed bandit or randomization approach?
Great read. I think the title "On Reddit, the earlier you comment, the louder your voice." or something to that tune would've made for a more impactful headline!
I think this analysis misses some subtlety of smaller subreddits by focusing on the huge mass subreddits. As such I don't know if the results are at all applicable for my experience when for example >80% of the threads currently on my frontpage have <30 comments total (and as such would be cut off from the analysis). And I don't think I'm such a total outlier here, the long tail of reddit is really long.
That's the problem with threaded views, as opposed to the flat-view style of other forums. Granted with the latter system, the first few comments on a new page will always get more exposure, too.
This is not a surprise - it is I guess inevitable and intuitively understood - I mean who bothers commenting on a thread here that has 500 earlier comments? There is zero chance of anyone finding reading or replying
I like the idea of less wrong using the right hand columns
Vote inflation might serve as an interesting dynamic for allowing other's voices to be heard. While a new vote will always carry the same value, acquired votes may suffer from inflationary decay, providing a rotation on viewed comments. You could probably modify this inflation based on thread velocity.
[+] [-] Houshalter|9 years ago|reply
Hacker news has a nice solution to this problem. New comments appear at the top of the thread, and then slowly fall unless they get votes. I think this works very well, although I think it's still a bit too far on the exploitation tradeoff.
I've thought about how to solve this problem, and I think the best solution is votes/time since the comment was posted. New comments will immediately appear at the top, since they have infinite vote/time ratio. But they will quickly fall to their correct ranking.
There are other solutions, like treating it like a multi-armed bandit problem. And you can use something like https://en.wikipedia.org/wiki/Thompson_sampling . Create a probability distribution for the upvote/downvote ratio for every comment. Sample from that distribution, and rank the comments accordingly. This should be close to optimal, but it requires people to use the downvote button lots (because it measures comment quality by upvote/downvote ratio, which is what reddit does currently.) Without downvotes, you could use time again, and create a probability distribution for votes/time.
[+] [-] cjlars|9 years ago|reply
To put it into economic terms, if users have high loss aversion (from being upset, etc.) relative to their thirst for knowledge gained from great comments, you would expect exploitation to be overweighted vs exploration. The business value maximizing solution would then be that once an appropriate top comment is found, the thread should tend to stay in that equilibrium.
[+] [-] DannyBee|9 years ago|reply
The traditional solution to this problem is wilson score intervals.
The real problem there is that most sites that use them only use one part of the bound, instead of using both. Ideally, you use the lower bound to scale how good something is, and the upper bound to determine with what priority you show it order to get more votes on it to determine better bounds to be more confident of your lower bound.
This is what Google Moderator did, and pretty much never had these problems as a result.
(Is it perfect? No. You need a good place/way to show things for voting. But assuming you get that, it works very well in practice).
If you only use the lower bound, you get systems where you only ever become more confident in the scores of the things that show up first (like is happening here).
[+] [-] sdegutis|9 years ago|reply
[+] [-] lotu|9 years ago|reply
Given all the worry we have about about how our online discussions work I think their is real societal (if not extrable monetary) value in discovering a better solution.
[+] [-] cousin_it|9 years ago|reply
[+] [-] nagvx|9 years ago|reply
(As an aside, it would be entertaining to take it to an extreme - seeing the top voted comments also shows you the bottom voted, most controversial also shows least, etc. A sort of anti-echo-chamber measure.)
[+] [-] cs702|9 years ago|reply
[+] [-] singham|9 years ago|reply
[+] [-] cs702|9 years ago|reply
I suspect HN is subject to the same "first comment effect," often to the detriment of later comments that might be more deserving of the top spot.
Does anyone here have ideas for fixing this?
[+] [-] pjc50|9 years ago|reply
[+] [-] mason240|9 years ago|reply
Have three categories: New, Best (up:down ratio is high), Controversial (up:down ratio of 1:1).
Pick one from each category and then randomly make them the top three. For larger threads you can salt them into the top 10 comments.
New comments that are truly good have a chance of being picked up to Best or Hot, and Controversial comments spur discussion and moderate the self-reinforcing circle-jerk that threads descend into.
[+] [-] joezydeco|9 years ago|reply
If someone like tptacek (with 259K karma) votes my comment up, I think that's a lot more relevant than someone with 300 karma.
[+] [-] david-given|9 years ago|reply
I actually think this would be an interesting experiment to try. Who's hacker friendly and has a well-used comment system that we know?
[+] [-] greggman|9 years ago|reply
[+] [-] no_protocol|9 years ago|reply
[+] [-] sotojuan|9 years ago|reply
[+] [-] malnourish|9 years ago|reply
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] gnicholas|9 years ago|reply
For example, if the first comment is roughly "I read the linked article and disagree because of A, B, and C" then someone with an opposing viewpoint would probably reply to the existing first comment instead of creating a separate comment.
This doesn't mean that the second commenter's voice isn't heard—it just means it all happens under the thread created by the first commenter.
[+] [-] kemitche|9 years ago|reply
The "best" sort, by the way, does not use time as a factor, but rather generates a confidence score. Newer comments have less votes and thus less confidence (which ends up giving them a slight boost).
[+] [-] asddddd|9 years ago|reply
A root-level comment when there are already dozens is unlikely to be noticed; there is significant momentum behind the top comments, usually for good reason - whether jokes, clever insight, or a popular viewpoint.
Instead, if you reply near the top in a context where it makes some sense, you skip the root-level graveyard and are almost guaranteed good placement with the downside of "overhead" due to the upwards chain always appearing first.
Consider also that the penalty for being the 2nd, 3rd, or worse comment at root-level gradually increases due to the increasingly nested replies to each highly rated comment. If you assume root-level comments express opinions, minority opinions are also more likely to maintain good position in a reply comment such as the one you posited.
Lastly, "trickle down karma" could be a factor. A quality reply to a root-level comment could enhance the parent comment's value and result in additional karma for them. Such situations could include biases like assuming comments which merited replies are interesting due to peer validation and thus reading them, and more fundamental added value such as uncited sources being given.
[+] [-] tgb|9 years ago|reply
[+] [-] minimaxir|9 years ago|reply
The subreddits not posted as exceptions in the article typically followed the global trend.
[+] [-] bagrow|9 years ago|reply
Edit: It's probably related to the cutoff of 30 comments, but it's still not obvious to me...
[+] [-] KayEss|9 years ago|reply
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] gggggggg|9 years ago|reply
Some of the best HN comments can come late and never make it to the top.
[+] [-] minimaxir|9 years ago|reply
I would perform the same analysis on HN data to confirm if I could. (HN does not expose comment scores)
[+] [-] baccheion|9 years ago|reply
You'd also have to factor in the likelihood of each person up/down voting (if it's viewed by many people who weren't going to vote anyway, then no votes are nothing out of the ordinary).
[+] [-] gregw134|9 years ago|reply
[+] [-] dd367|9 years ago|reply
[+] [-] zokier|9 years ago|reply
[+] [-] dmfdmf|9 years ago|reply
[+] [-] et-al|9 years ago|reply
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] lifeisstillgood|9 years ago|reply
I like the idea of less wrong using the right hand columns
[+] [-] nix0n|9 years ago|reply
[+] [-] minimaxir|9 years ago|reply
[+] [-] miguelrochefort|9 years ago|reply
[+] [-] Chronic9g|9 years ago|reply
[deleted]
[+] [-] andrewmcwatters|9 years ago|reply