I worked at Facebook for some time, and did a bunch of data work. We had this culture of building something, looking at the results between the experiment groups, and then choosing the statistically more successful one -- i.e. a newsfeed algorithm that had better engagement.
This sounds great at first, and certainly is straightforward if you want a promotion. But behind the scenes some of us had this thought that our observations only amounted to short-term gains. Although we had small long-term experiment holdout groups, the truth is they were rarely reviewed because it was unsexy.
My current thinking is that features like the echo chamber effects from Facebook's algorithms, Snapchat's snap streaks, and clickbait like this, all serve to optimize short-term engagement. Yeah, I want to watch that sexy new show or keep my streak going or have my opinions validated. But there's a diminishing return on clickbait, hollow articles isn't there? I can only fill up so much time with garbage like that before I'm bored. I can only like so many posts before I feel like they're all the same. And once my snap streak is broken I hate snapping.
The data/engineering/product loops at tech companies favor boosting short-term metrics; The employees are incentivized to do so and this is what they measure, so this is what they build. That's why we end up with features like this. That's why Snapchat fell off. That's why Facebook fell off. And that's why Netflix feels increasingly stale (despite there being a lot of quality content if you dig).
Humans are instinctively drawn to a lot of things that are really bad for us in significant amounts, because we are designed to fit into an environment where those things are scarce. Sugar, fat, inactivity, "interesting news", outrage. These all steal our attention because that response helped our ancestors survive and pick out these rare but important treats. But in typical human fashion we've now crafted a world that gives us these things all the time, and it's making us sick and miserable as a result.
"But we're just giving the people what they want" some might say. Well, depends on the definition of "want". In a taste test, junk food would win over broccoli for me, hands down - but it's broccoli that ends up on my table more often than not, because it's what makes me happier and healthier in the long run. It wasn't until my thirties before I realized just how sluggish I got after eating junk food. But if Facebook was running our diets, their algorithm would long since have "optimized" its way to junk food for all of us.
Sometimes the worst thing you can do is give people exactly what they ask for. Being healthy in a world like the one we've created requires much, much more restraint and self-discipline than it used to. "The algorithm" is basically the digital incarnation of the little devil on our shoulder whispering that we should treat ourselves.
> The data/engineering/product loops at tech companies favor boosting short-term metrics
I think this hits the nail on the head. Leadership is hard, and executives can be many layers removed from the frontline. It seems that A/B tests and family have proven a really effective vehicle to communicate across leadership levels what has happened, why it happened, and the impact it had on the business. I'm definitely guilty as charged of having participated in this "pyramid scheme".
Teams get a tool to structure work and determine success/fail. Managers get a tool to detail exact impact to leadership on how they're solving business problems. Executives get a tool showing how teams are being deployed on thoughtful projects with an iterative and rigorous scientific approach that helps increase confidence around deploying capital. The board gets some slides during the quarterly meeting and maybe might even see some charts showing growth and less cash burn.
Eventually over longer periods like a year, executives will notice different trends than what the short-term numbers are showing and will have to make real hard decisions about how to steer the ship.
They don't seem to have fallen off, are still the most widely used platforms in the world with billions of users in total. It is popular to think that A/B testing leaves you in local maxima, but what about alternatives? The only thing I've seen work is something like Pixar's brain trust - you do testing but you also rely on the good judgement of a small group of people with a strong sense of vision. Judging good judgement is pretty hard though, and you won't know if they're just full of it or winging it unless you try and trust. And so everyone does A/B testing which to be honest sounds much better than your average PM making decisions that are pulled out of their ass.
This is the consequence of lack of vision. What is Netflix's long term goal? Ideally, it would be to fund and broadcast fantastic cinema. Yet, I really don't get the impression that contributing to the art form of film is very high up on the metrics chart.
As another former Facebook employee, I mostly agree with this.
One nitpick is that although you may have not looked at the holdouts often, somebody definitely did. Different teams use their holdouts differently, but leadership probably looked at your holdouts at least a few times per quarter. And now with recent infra and changes, it's very likely that data scientists or product managers somewhere in your org are responsible for explaining results to higher ups at least quarterly.
All that is to say, the cause isn't that Facebook (and Netflix) aren't thinking about or monitoring things in the long term. It's that they are measuring the wrong things, because it's very hard to measure things like "this is clickbait".
Yes, Facebook is now almost useless. The only reason to be there is to keep track of friends and family, but it has gotten to the point where if a family member has a big life event, I'm hearing about it through traditional in-person or on-voice-call word of mouth _faster_ than through Facebook, because Facebook keeps deciding for me what it wants me to see.
Netflix has kept trying to make it harder for me to find what I want for the last few years, and I use it less and less, and now pay for fewer accounts than in the past. They are in danger of losing customers, with their annoying mouseover/autoplay/clickbait interface.
Yes, I'd love to see a more robust approach to novelty decay rates (to make up a metric) in A/B testing, especially around engagement.
You'd think that would be of interest to the business to know that X "successful" intervention had a typical average lifetime of Y before reverting to the mean, for example.
Then again, there's the idea that competitive advantage for social networks is just finding enough novel interventions before the novelty of the platform itself is exhausted, and FB has no short supply of other platforms to milk in that regard.
Netflix has somehow conditioned me to expect disappointment in movies with interesting looking cover art, which I now actively ignore, and don't even click through for descriptions anymore.
"Huh, that looks interesting. Not falling for that old trick again! What's that half-off-screen ambiguous cover on the next row down?"
I think you can maintain the feedback loop without degrading the user experience if you actually ask them what they want to be shown.
The problem lies in that "engagement" definition : if the user clicked on a thumb, then they must have been attracted to it. In reality, there is considerably more explanation for that click : They could be looking for a specific title, mistakenly thought _this_ actor was playing in the movie, clicked on it to reset the search, wondered if this was a show or a film, missclicked, just picked the least annoying in an ocean of shitty propositions... An algorithm alone can never find its way for better suggestions if it lacks _intent_. The outcome of this lack of data and the lack of data scientists' imagination, will always be stuck in a local minima.
So, what solution do you have to gather this precious intent and act on it ? Well, you start small on predictable things : "what would you recommend to Agatha if they liked _Expendables_ ?" "Is _RED_ a good addition to the list if I'm looking for Action/Comedy films ?"
And later "Are you looking for Action/Comedy films ? here is some proposals"
It works the same for different approaches, for picking thumbnails, or promoting movies. Just ask user for what's motivate them. If Netflix had asked me last week, I would have told them their catalog is getting worse and worse so we prefer using disney+ instead.
i do not understand snapchat and their business model. they started out as a sexting app that deletes the photos and that is where it remained for a long time. i remember news of how "deleted snapchat photos can be undeleted" tutorials and i tried to investigate for fun but i could not understand the UI back then and the tutorials didnt work so i dropped it.
Apparently its pretty big these days in the kids and they have "streaks" as you mention. i hear multiple hundred thousand streaks and other things but to what end?
how is snapchat justifying the storage, bandwidth and processing of data? i mean facebook has ads which they earn, same for twitter but what about snapchat? do they have a sustainable revenue stream because no where i heard snapchat had a paid option.
reddit was user funded for a long time with daily goals and all that so that was not an issue mostly but what snapchat done to address that.
snapchat is all the rage in my city, kids who are now on online classes send snaps because i hear about it. everyone knows how to make that twistface pout or whatever but why?
genuinely curious about it
It seems even worse than that. During my stints in social media, there would be ‘research projects’ that were simply fishing expeditions for meaningful data. Something like: which user demographics have a correlation to behavior x. And then you’d just search and search and search and hope you found something. If you didn’t find something, then you’d be encouraged by a team level manager to adjust or tweak some approach to take advantage of volatility to force the correlation. Maybe you shift the sample start date, maybe you skip a normalization dimension.
The directors would ask about things like normalization and if things were measured over a ncertain time period, but if you had some excuse or answer prepared, it was enough to get a nod and some recognition for your presentation. Heck, sometimes your team manager would do the presentation on your behalf to lay claim to some of that precious impact.
Then, after the new review cycle starts, priorities change and you never have to follow up on the holes in your data. Or, you can even admit it has holes, you already got credit.
> The employees are incentivized to do so and this is what they measure, so this is what they build
I also work in data teams and this is directly my observation too.
Closely related are Conway's Law (you ship your org chart) and Goodhart's Law (commonly paraphrased as 'When a measure becomes a target, it ceases to be a good measure.')
In Netflix's case I certainly feel the shipped product is too close of a reflection of org structure and incentives and user experience suffers as a result.
This feels particularly evident in the treatment of thumbnails and title descriptions. Often I find myself clicking through on a thumbnail in order to read the description only to find that the I've already read the description and didn't want to watch that title, but the thumbnail has since changed.
I'm sure there is some team in Netflix whose sole purpose is increasing thumbnail clickthrough rates. And they are probably succeeding in that respect by changing the thumbnails. They get to win at their portion of some funnel, even if the net result is a lousy user experience.
>> But behind the scenes some of us had this thought that our observations only amounted to short-term gains.
I'm convinced this must be the case at Facebook. I used to love the newsfeed but it has steadily declined so much for me that I stopped using it. It used to highlight a wide variety of friends updates. Then, only starred "close friends" and now mostly junk news items. Amazing things happen for friends (weddings, births) and I dont hear about it (no they aren't blocking me, i've confirmed).
I can to individually to each person's profile and see everything, but how realistic is that.
I'm sure there was a short boost as people checked more and more to news that didn't appear. Now, you check five times, you get the same 30 newsfeed items over and over and then "you're all caught up".
I can totally see the short term boost and long term decline with such a strategy.
Online Analytics is now really nuanced - you need to know what metrics are important for * your * business, not just use boilerplate kpis.
For instance, an e-commerce website is clearly looking to lead the user to a lot purchase and the more they purchase, it’s good for the business so kpis around sales conversion help and recommenders help your business increase sales.
For Netflix though, the users have already paid for the service after which they land on the website. Most users I imagine then expect to be provided all that Netflix has to offer in an easy way. So if I was a Netflix product owner, I’d be more interested in Kpis around search-ability, having an anti algorithm that “suggests” completely random obscure shows, “switchability” of users - how less of a time do users spend on a movie or show.
I imagine they’re doing this but as a user I don’t see this at least - they show the same old stale recommendations for me, I’m always trying to hack their search to find what I want and they continue to invest in content that’s mostly miss than hit. I wish they at least had a directory for me to browse through (at least I’ll be driving their engagement metrics to help them drive their valuations)
> i.e. a newsfeed algorithm that had better engagement.
Sometimes (but rarely) I click on something because I ask myself "Why would these a---- show me this? I want to know more about what's behind it in order to understand what could have made their algorithm chose me as a candidate for this content". Or to check where the scam-URL would lead me to, because this also happens, that ads sold on Google and Facebook are scams, links leading to malware.
Congratulations to them, if they think that this is the kind of engagement which should be valued, that any kind of engagement is a good and healthy engagement. These companies are so rotten and their engineers just don't care.
I’ve worked on similar experiments (starting about Stories) and we included long-term holdout experiment to measure the compounded impact of sharing. The fear at the time was more about professional content (with millions of views) vs. your friends. The worry was because people engaged more with the former but posted more if they saw the later.
I left before conclusions were drawn, but (according to press reports) those experiments changed the goals to more leading indicator of long-term trend, like posting rather than Likes. I joined again more recently and expected to see that it influenced the company.
Off topic: To me they seem to focus on quantity over quality. Can you recommend some of that quality content? I always quit after browsing their catalogue for 20 minutes or so.
The algorithm seems to have picked up the fact that I like watching SciFi and Zombie movies, so my Netflix homepage was this dark place filled with Sci Fi and Zombie movies.
One problem with this is that Netflix only has a handful of good SciFi and Zombie movies, and I've seen them all. So the homepage was filled with movies I've seen already, or the dime-a-dozen copycat movies that just rehash some ideas from popular movies in a slightly different way.
The much bigger problem is that even though I like watching Zombie movies, I actually enjoy lots of different movies. But somehow the Netflix algorithm only ever shows me this one genre.
So I cancelled my Netflix subscription, and went back to occasionally renting a film on iTunes or Amazon. I watch less now, but I end up watching more diverse and more interesting films.
> So I cancelled my Netflix subscription, and went back to occasionally renting a film on iTunes or Amazon.
I did that too, and then went one step further, because some fifteen years down the line, online video rental still hilariously, bafflingly sucks.
Twelve, thirteen years ago I was still renting physical DVDs. Back then, renting a physical item that was produced halfway around the world, shipped to Germany and distributed by a company with hundreds of physical locations staffed by employees was around two and a half times cheaper than downloading a file via iTunes. Apart from inflation, those prices never came down.
Adding insult to injury, the more expensive download is almost always worse:
On the majority of DVDs I get both the original and the dubbed voice track, plus subtitles in English and German.
That's important to me, because my partner vastly prefers either the dubbed version, or at least the original with German subtitles. When I watch something for myself, I vastly prefer the original - sometimes with English subtitles.
In online video rental (or purchase), I can often only get the dubbed version. If the original is available at all, it is sometimes another item to be bought separately. Either case almost never features both German and English subtitles.
And as the icing on the cake, to this day not all product pages on iTunes even _list_ the featured languages of a download, let alone their subtitle languages.
Combine that with all the other indignities of buffet streaming, such as titles being constantly rotated out or the incessant advertising on Amazon before every episode.
So — I've gone back to buying used DVDs. And since they have their own problems such as unskippables and horrible menus, I'm currently looking into building a NAS and will be ripping them into a personal media collection sometime in the future.
And then I will have come back full circle to 2005.
I've found that when I watch a Youtube video, at least half or more of the recommended videos on the right are videos that I've already seen before, often music. Does anyone know if there is way to turn this off so it can recommend me strictly new videos? It's infuriating.
The best guess for why this is that I've seen online is that it's because they're targeting younger age brackets more aggressively and kids love to rewatch the same things over and over.
+1 on renting films with some level of purpose. A lot of people I know refuse to pay for movie rentals.
I've seen this too many times: endless scrolling to try to find that one decent movie that comes free with the subscription.
But movie studios aren't stupid. They don't just give away their best movies for free. So those who simply aren't willing to pay are left watching Hitman or The Quake (fine Hulu content).
The price of movie rentals might seem high, but life is far too short to waste it watching things you don't even like that much.
Also: iTunes in particular happens to have very decent staff recommendations (i.e., actual human curation). I wouldn't be surprised if movie rental services like iTunes understand that their customers are looking for quality and not quantity, otherwise those customers would be on the streaming services.
I'm in the same boat as you. My question is, does this hyperoptimisation of preferences actually work for anyone other than children? I guess Netflix thinks it does.
When I found myself spending 15+ minutes looking for content and not finding anything, I just cancelled.
I just don't see the value in Netflix anymore. Everyone has wisened up to the fact that creating your own streaming service is extremely important - especially when you consider ROI.
So what differentiates streaming services is the catalog. Netflix now have to compete on the quality of their originals. They have some good shows once in a while but the their lead is not groundbreaking at all anymore.
I have a similar problem with amazon music and spotify: Yes, they are good at picking music I like, but really bad at adding something new.
Spotify is better at this, as it does not give you the one "your radio".
At amazon music it basically played only those songs I told it about, and a few very very narrow matches, but close to nothing I did not already know.
> users consider each title for a whopping 1.8 seconds
Users are most likely not “considering” them at all. We’re not machines going over one tile at a time and generating a score. Most of the time you’re looking for something specific and just trying to find it.
This type of metric is the worst. I really hope this data-driven fad dies down and we start designing with human factors in mind again.
Would Netflix have to do this if it actually had content people wanted to watch? It's just sad watching it double down on this pulp because every other copyright holder took their ball and went home.
The way Netflix presents thumbnail images custom tailored to the user is ingenious. There is a lot of insight there on how people choose content based on preferences for star actors, subject matter, tone etc.
Personally however, as an avid film viewer, this massive algorithmic curation is completely not for me.
A cool example and complete opposite of the Netflix approach is the Mubi approach. There, the focus is not on giving me exactly what I think I want, but instead on offering this narrow curated selection along with content written by actual film critics. As a result I watch things that I did not expect.
This curation aspect is something Netflix strategically completely opted out of. And this makes sense - their goal is to have an active subscriber base and achieving that goal doesn't factor in the existing film/tv culture.
Personally although I do see “sexy” titles on Netflix strewn about, it isn’t a big distraction or problem the way this article makes it out to be. The real problem is that most of the content is B tier straight to video content. I’m drowning in choices and I don’t mean within Netflix but elsewhere - I’m simply not compelled to spend my valuable minutes there and these days I rarely load their app. I’ll probably end up cancelling it, as my household has increasingly returned to analog entertainment (books, conversations) over the pandemic, with other viewing time going to content we deliberately seek out rather than content we casually surf. At one point I was hopeful Netflix could produce first party content that was great. But each promising attempt (for example House of Cards, Marco Polo, or Altered Carbon) fizzles and gets cancelled after a season or two. Today I can’t see myself investing my time into their new content because I don’t want to be disappointed by their eventual cancellation, and I am left wondering who their service is for.
I really hate it that when I'm digesting an episode of Rick and Morty I have to scramble for the remote or I'm watching some completely uninteresting series about a family in the 19th century or something. I just want the end screen and sounds to finish and then turn off the telly and go to bed. In stead I feel slapped in the face.
Consistently my watching experience ends with (increasingly large) negative feelings. How can that be a good choice for a company offering watching experiences?
I believe Netflix has talked about thumbnail experimentation at length in some of their engineering blogs [1]. To me, it seems perhaps Netflix has figured out a running theme in the author's viewing habits... perhaps the scandalous, soapy stuff is what they keep clicking.
Same kind of reaction I get when a guy friend tells me they keep getting ads for women's bikinis or women's underwear in their Instagram ads. It isn't sexualization of Instagram... it is a reflection of your interaction with the platform!
Movie posters have used salacious and sensational words and imagery for as long as they've existed. If anything, Netflix titles have been slowly working toward parity to that standard, as opposed to devolving to actual clickbait.
Let's be real about Netflix: after looking like it was going to disrupt the media conglomerates, I now think that all it has managed to do is wake the sleeping giants. Long-term, I am bearish on Netflix. I would go as far to say that Netflix is slow-declining its way into being eventually acquired by a member of the big six.
Here are the issues:
First: it's basically the most expensive streaming service, topping out at $18/month.
Hulu's most expensive plan is $12. Discovery+ is $7. Disney+ is $8. HBO Max is $15. Paramount+ is $10. (All prices ad-free plans)
I think an argument could be made that all or nearly all of those services are offering a better content library at a lower price when compared to Netflix.
Discovery+ especially...holy hell if you are into reality shows it's endless. And it's $7. I would pay $7/month just for access to every House Hunters episode imaginable without ads, lo and behold my dream came true.
Netflix is doing this clickbait stuff because their content sucks. Clickbait is what you do when your content doesn't speak for itself.
Sure, every content business has to make a "headline" to draw your attention. But when you see a "clickbait" headline in something like The New York Times you know you're being drawn into something that can be potentially rich in effort, and therefore the term "clickbait" doesn't really apply. At least there's an article behind the hook. "Clickbait" more specifically means you're being tricked into visiting something that everyone knows definitely sucks, including its creator.
Netflix knows their content sucks, and I'm not sure they care or can think of a viable business model to improve it.
Here’s what I don’t understand (either about the premise of the article if incorrect or Netflix as a business if correct): why does Netflix need to drive proactive short-term engagement? They don’t make money from ads. If I’m subscribed (I’m not anymore) then their business is accounted for. I’m subscribed because they’re a better platform than TV and when I want to watch a show they have, I will. They’re getting my money regardless and they aren’t getting more if it the more I watch. Maybe they really feel pressure from other streaming platforms?
If I search for "streetcar named desire" I want a finder which says "nope, but here's the other work by {Elia Kazan, Tenesee Williams, Marlon Brando, Carl Malden, Jessica Tandy, Kim Hunter}
Oddly, I don't want "here's the abortive experimental 1970s film shot in Russia by a drug crazed student also called "streetcar"
But.. from Netflix's PoV, its equally likely I did want that because they know I watch e.g. Luis Bunel films
So.. how can they know? Answer: they can't. They simply can't get my mood right, all the time. Sometimes, they will do well and guess. Sometimes, they do really badly.
Another take: How many fans of the british "Office" wanted to be told to watch the American "Office" ?
Anyone have a good website that lets you find netflix shows? I like a wide variety of stuff and I think Spotify/Netflix/Youtube always gets too confused to be useful.
Netflix has also become unbearably "woke". I've simply given up on Netflix for that reason but also due to many other reasons like the what is described here.
After reading the article on my phone and hitting the back button to get back here, I'm instead presented with a screen showing more content and pleading me to stay on the site (Keep on reading!). Talk about the kettle calling the pot black.
It's interesting, because one of the earliest lessons I was taught in college was that greedy algorithms tend to lead to sub-optimal results.
Is the problem that they're not teaching that in college anymore, or is it that the people in charge of these things are non-engineers who don't necessarily even know what a greedy algorithm is, let alone that the principle might apply to business practices and not just software implementations.
[+] [-] personjerry|4 years ago|reply
This sounds great at first, and certainly is straightforward if you want a promotion. But behind the scenes some of us had this thought that our observations only amounted to short-term gains. Although we had small long-term experiment holdout groups, the truth is they were rarely reviewed because it was unsexy.
My current thinking is that features like the echo chamber effects from Facebook's algorithms, Snapchat's snap streaks, and clickbait like this, all serve to optimize short-term engagement. Yeah, I want to watch that sexy new show or keep my streak going or have my opinions validated. But there's a diminishing return on clickbait, hollow articles isn't there? I can only fill up so much time with garbage like that before I'm bored. I can only like so many posts before I feel like they're all the same. And once my snap streak is broken I hate snapping.
The data/engineering/product loops at tech companies favor boosting short-term metrics; The employees are incentivized to do so and this is what they measure, so this is what they build. That's why we end up with features like this. That's why Snapchat fell off. That's why Facebook fell off. And that's why Netflix feels increasingly stale (despite there being a lot of quality content if you dig).
[+] [-] m12k|4 years ago|reply
"But we're just giving the people what they want" some might say. Well, depends on the definition of "want". In a taste test, junk food would win over broccoli for me, hands down - but it's broccoli that ends up on my table more often than not, because it's what makes me happier and healthier in the long run. It wasn't until my thirties before I realized just how sluggish I got after eating junk food. But if Facebook was running our diets, their algorithm would long since have "optimized" its way to junk food for all of us.
Sometimes the worst thing you can do is give people exactly what they ask for. Being healthy in a world like the one we've created requires much, much more restraint and self-discipline than it used to. "The algorithm" is basically the digital incarnation of the little devil on our shoulder whispering that we should treat ourselves.
[+] [-] tomnipotent|4 years ago|reply
I think this hits the nail on the head. Leadership is hard, and executives can be many layers removed from the frontline. It seems that A/B tests and family have proven a really effective vehicle to communicate across leadership levels what has happened, why it happened, and the impact it had on the business. I'm definitely guilty as charged of having participated in this "pyramid scheme".
Teams get a tool to structure work and determine success/fail. Managers get a tool to detail exact impact to leadership on how they're solving business problems. Executives get a tool showing how teams are being deployed on thoughtful projects with an iterative and rigorous scientific approach that helps increase confidence around deploying capital. The board gets some slides during the quarterly meeting and maybe might even see some charts showing growth and less cash burn.
Eventually over longer periods like a year, executives will notice different trends than what the short-term numbers are showing and will have to make real hard decisions about how to steer the ship.
It's a hell of a hamster wheel.
[+] [-] vasco|4 years ago|reply
[+] [-] keiferski|4 years ago|reply
[+] [-] mgraczyk|4 years ago|reply
One nitpick is that although you may have not looked at the holdouts often, somebody definitely did. Different teams use their holdouts differently, but leadership probably looked at your holdouts at least a few times per quarter. And now with recent infra and changes, it's very likely that data scientists or product managers somewhere in your org are responsible for explaining results to higher ups at least quarterly.
All that is to say, the cause isn't that Facebook (and Netflix) aren't thinking about or monitoring things in the long term. It's that they are measuring the wrong things, because it's very hard to measure things like "this is clickbait".
[+] [-] Vrondi|4 years ago|reply
Netflix has kept trying to make it harder for me to find what I want for the last few years, and I use it less and less, and now pay for fewer accounts than in the past. They are in danger of losing customers, with their annoying mouseover/autoplay/clickbait interface.
[+] [-] lukestevens|4 years ago|reply
You'd think that would be of interest to the business to know that X "successful" intervention had a typical average lifetime of Y before reverting to the mean, for example.
Then again, there's the idea that competitive advantage for social networks is just finding enough novel interventions before the novelty of the platform itself is exhausted, and FB has no short supply of other platforms to milk in that regard.
[+] [-] beerandt|4 years ago|reply
"Huh, that looks interesting. Not falling for that old trick again! What's that half-off-screen ambiguous cover on the next row down?"
[+] [-] Fiahil|4 years ago|reply
The problem lies in that "engagement" definition : if the user clicked on a thumb, then they must have been attracted to it. In reality, there is considerably more explanation for that click : They could be looking for a specific title, mistakenly thought _this_ actor was playing in the movie, clicked on it to reset the search, wondered if this was a show or a film, missclicked, just picked the least annoying in an ocean of shitty propositions... An algorithm alone can never find its way for better suggestions if it lacks _intent_. The outcome of this lack of data and the lack of data scientists' imagination, will always be stuck in a local minima.
So, what solution do you have to gather this precious intent and act on it ? Well, you start small on predictable things : "what would you recommend to Agatha if they liked _Expendables_ ?" "Is _RED_ a good addition to the list if I'm looking for Action/Comedy films ?"
And later "Are you looking for Action/Comedy films ? here is some proposals"
It works the same for different approaches, for picking thumbnails, or promoting movies. Just ask user for what's motivate them. If Netflix had asked me last week, I would have told them their catalog is getting worse and worse so we prefer using disney+ instead.
[+] [-] 2Gkashmiri|4 years ago|reply
how is snapchat justifying the storage, bandwidth and processing of data? i mean facebook has ads which they earn, same for twitter but what about snapchat? do they have a sustainable revenue stream because no where i heard snapchat had a paid option.
reddit was user funded for a long time with daily goals and all that so that was not an issue mostly but what snapchat done to address that.
snapchat is all the rage in my city, kids who are now on online classes send snaps because i hear about it. everyone knows how to make that twistface pout or whatever but why? genuinely curious about it
[+] [-] antiterra|4 years ago|reply
The directors would ask about things like normalization and if things were measured over a ncertain time period, but if you had some excuse or answer prepared, it was enough to get a nod and some recognition for your presentation. Heck, sometimes your team manager would do the presentation on your behalf to lay claim to some of that precious impact.
Then, after the new review cycle starts, priorities change and you never have to follow up on the holes in your data. Or, you can even admit it has holes, you already got credit.
[+] [-] pdinny|4 years ago|reply
I also work in data teams and this is directly my observation too.
Closely related are Conway's Law (you ship your org chart) and Goodhart's Law (commonly paraphrased as 'When a measure becomes a target, it ceases to be a good measure.')
In Netflix's case I certainly feel the shipped product is too close of a reflection of org structure and incentives and user experience suffers as a result.
This feels particularly evident in the treatment of thumbnails and title descriptions. Often I find myself clicking through on a thumbnail in order to read the description only to find that the I've already read the description and didn't want to watch that title, but the thumbnail has since changed.
I'm sure there is some team in Netflix whose sole purpose is increasing thumbnail clickthrough rates. And they are probably succeeding in that respect by changing the thumbnails. They get to win at their portion of some funnel, even if the net result is a lousy user experience.
[+] [-] TuringNYC|4 years ago|reply
I'm convinced this must be the case at Facebook. I used to love the newsfeed but it has steadily declined so much for me that I stopped using it. It used to highlight a wide variety of friends updates. Then, only starred "close friends" and now mostly junk news items. Amazing things happen for friends (weddings, births) and I dont hear about it (no they aren't blocking me, i've confirmed).
I can to individually to each person's profile and see everything, but how realistic is that.
I'm sure there was a short boost as people checked more and more to news that didn't appear. Now, you check five times, you get the same 30 newsfeed items over and over and then "you're all caught up".
I can totally see the short term boost and long term decline with such a strategy.
[+] [-] anon9001|4 years ago|reply
Don't sell yourself short! You're surrounded by tons of people that never get their fill of content. Just relax and let it happen ;)
[+] [-] roystonvassey|4 years ago|reply
Online Analytics is now really nuanced - you need to know what metrics are important for * your * business, not just use boilerplate kpis.
For instance, an e-commerce website is clearly looking to lead the user to a lot purchase and the more they purchase, it’s good for the business so kpis around sales conversion help and recommenders help your business increase sales.
For Netflix though, the users have already paid for the service after which they land on the website. Most users I imagine then expect to be provided all that Netflix has to offer in an easy way. So if I was a Netflix product owner, I’d be more interested in Kpis around search-ability, having an anti algorithm that “suggests” completely random obscure shows, “switchability” of users - how less of a time do users spend on a movie or show.
I imagine they’re doing this but as a user I don’t see this at least - they show the same old stale recommendations for me, I’m always trying to hack their search to find what I want and they continue to invest in content that’s mostly miss than hit. I wish they at least had a directory for me to browse through (at least I’ll be driving their engagement metrics to help them drive their valuations)
[+] [-] qwertox|4 years ago|reply
Sometimes (but rarely) I click on something because I ask myself "Why would these a---- show me this? I want to know more about what's behind it in order to understand what could have made their algorithm chose me as a candidate for this content". Or to check where the scam-URL would lead me to, because this also happens, that ads sold on Google and Facebook are scams, links leading to malware.
Congratulations to them, if they think that this is the kind of engagement which should be valued, that any kind of engagement is a good and healthy engagement. These companies are so rotten and their engineers just don't care.
[+] [-] bertil|4 years ago|reply
I left before conclusions were drawn, but (according to press reports) those experiments changed the goals to more leading indicator of long-term trend, like posting rather than Likes. I joined again more recently and expected to see that it influenced the company.
I’m not working there at the moment.
[+] [-] dangus|4 years ago|reply
If you have a nice experience for a week or two and the price is low enough you'll probably never watch Netflix (e.g., ME) and keep on paying for it.
[+] [-] aeoleonn|4 years ago|reply
It will lead to the end of Facebook and hopefully these practices you’ve described... once the lesson is learned.
The end of Facebook, Snap, and other scientifically-designed-to-be-attention-exploiting apps/sites would be very positive.
[+] [-] uxcolumbo|4 years ago|reply
[+] [-] dreamer7|4 years ago|reply
[+] [-] omniscient_oce|4 years ago|reply
[+] [-] yarcob|4 years ago|reply
The algorithm seems to have picked up the fact that I like watching SciFi and Zombie movies, so my Netflix homepage was this dark place filled with Sci Fi and Zombie movies.
One problem with this is that Netflix only has a handful of good SciFi and Zombie movies, and I've seen them all. So the homepage was filled with movies I've seen already, or the dime-a-dozen copycat movies that just rehash some ideas from popular movies in a slightly different way.
The much bigger problem is that even though I like watching Zombie movies, I actually enjoy lots of different movies. But somehow the Netflix algorithm only ever shows me this one genre.
So I cancelled my Netflix subscription, and went back to occasionally renting a film on iTunes or Amazon. I watch less now, but I end up watching more diverse and more interesting films.
[+] [-] janfoeh|4 years ago|reply
I did that too, and then went one step further, because some fifteen years down the line, online video rental still hilariously, bafflingly sucks.
Twelve, thirteen years ago I was still renting physical DVDs. Back then, renting a physical item that was produced halfway around the world, shipped to Germany and distributed by a company with hundreds of physical locations staffed by employees was around two and a half times cheaper than downloading a file via iTunes. Apart from inflation, those prices never came down.
Adding insult to injury, the more expensive download is almost always worse:
On the majority of DVDs I get both the original and the dubbed voice track, plus subtitles in English and German.
That's important to me, because my partner vastly prefers either the dubbed version, or at least the original with German subtitles. When I watch something for myself, I vastly prefer the original - sometimes with English subtitles.
In online video rental (or purchase), I can often only get the dubbed version. If the original is available at all, it is sometimes another item to be bought separately. Either case almost never features both German and English subtitles.
And as the icing on the cake, to this day not all product pages on iTunes even _list_ the featured languages of a download, let alone their subtitle languages.
Combine that with all the other indignities of buffet streaming, such as titles being constantly rotated out or the incessant advertising on Amazon before every episode.
So — I've gone back to buying used DVDs. And since they have their own problems such as unskippables and horrible menus, I'm currently looking into building a NAS and will be ripping them into a personal media collection sometime in the future.
And then I will have come back full circle to 2005.
[+] [-] Gys|4 years ago|reply
I cannot believe I have to manually 'find' those movies.
My default homepage always seems to show the same movies. Many of them I saw already and others that I do not want to waste time watching.
[+] [-] omniscient_oce|4 years ago|reply
The best guess for why this is that I've seen online is that it's because they're targeting younger age brackets more aggressively and kids love to rewatch the same things over and over.
[+] [-] dangus|4 years ago|reply
I've seen this too many times: endless scrolling to try to find that one decent movie that comes free with the subscription.
But movie studios aren't stupid. They don't just give away their best movies for free. So those who simply aren't willing to pay are left watching Hitman or The Quake (fine Hulu content).
The price of movie rentals might seem high, but life is far too short to waste it watching things you don't even like that much.
Also: iTunes in particular happens to have very decent staff recommendations (i.e., actual human curation). I wouldn't be surprised if movie rental services like iTunes understand that their customers are looking for quality and not quantity, otherwise those customers would be on the streaming services.
[+] [-] peanut_worm|4 years ago|reply
[+] [-] joe_fishfish|4 years ago|reply
[+] [-] omegalulw|4 years ago|reply
I just don't see the value in Netflix anymore. Everyone has wisened up to the fact that creating your own streaming service is extremely important - especially when you consider ROI.
So what differentiates streaming services is the catalog. Netflix now have to compete on the quality of their originals. They have some good shows once in a while but the their lead is not groundbreaking at all anymore.
[+] [-] maverwa|4 years ago|reply
[+] [-] dannyr|4 years ago|reply
The most common thumbnail I see is a picture of someone looks like they are blown away with their hands on their head and their mouth wide open.
[+] [-] ricardobeat|4 years ago|reply
Users are most likely not “considering” them at all. We’re not machines going over one tile at a time and generating a score. Most of the time you’re looking for something specific and just trying to find it.
This type of metric is the worst. I really hope this data-driven fad dies down and we start designing with human factors in mind again.
[+] [-] matheusmoreira|4 years ago|reply
[+] [-] scyzoryk_xyz|4 years ago|reply
Personally however, as an avid film viewer, this massive algorithmic curation is completely not for me.
A cool example and complete opposite of the Netflix approach is the Mubi approach. There, the focus is not on giving me exactly what I think I want, but instead on offering this narrow curated selection along with content written by actual film critics. As a result I watch things that I did not expect.
This curation aspect is something Netflix strategically completely opted out of. And this makes sense - their goal is to have an active subscriber base and achieving that goal doesn't factor in the existing film/tv culture.
[+] [-] throwawaysea|4 years ago|reply
[+] [-] teekert|4 years ago|reply
Consistently my watching experience ends with (increasingly large) negative feelings. How can that be a good choice for a company offering watching experiences?
[+] [-] darkwizard42|4 years ago|reply
Same kind of reaction I get when a guy friend tells me they keep getting ads for women's bikinis or women's underwear in their Instagram ads. It isn't sexualization of Instagram... it is a reflection of your interaction with the platform!
[1: https://netflixtechblog.com/selecting-the-best-artwork-for-v...]
[+] [-] robbedpeter|4 years ago|reply
[+] [-] dangus|4 years ago|reply
Here are the issues:
First: it's basically the most expensive streaming service, topping out at $18/month.
Hulu's most expensive plan is $12. Discovery+ is $7. Disney+ is $8. HBO Max is $15. Paramount+ is $10. (All prices ad-free plans)
I think an argument could be made that all or nearly all of those services are offering a better content library at a lower price when compared to Netflix.
Discovery+ especially...holy hell if you are into reality shows it's endless. And it's $7. I would pay $7/month just for access to every House Hunters episode imaginable without ads, lo and behold my dream came true.
Netflix is doing this clickbait stuff because their content sucks. Clickbait is what you do when your content doesn't speak for itself.
Sure, every content business has to make a "headline" to draw your attention. But when you see a "clickbait" headline in something like The New York Times you know you're being drawn into something that can be potentially rich in effort, and therefore the term "clickbait" doesn't really apply. At least there's an article behind the hook. "Clickbait" more specifically means you're being tricked into visiting something that everyone knows definitely sucks, including its creator.
Netflix knows their content sucks, and I'm not sure they care or can think of a viable business model to improve it.
[+] [-] dcow|4 years ago|reply
[+] [-] ggm|4 years ago|reply
Oddly, I don't want "here's the abortive experimental 1970s film shot in Russia by a drug crazed student also called "streetcar"
But.. from Netflix's PoV, its equally likely I did want that because they know I watch e.g. Luis Bunel films
So.. how can they know? Answer: they can't. They simply can't get my mood right, all the time. Sometimes, they will do well and guess. Sometimes, they do really badly.
Another take: How many fans of the british "Office" wanted to be told to watch the American "Office" ?
[+] [-] zz865|4 years ago|reply
[+] [-] sgt|4 years ago|reply
[+] [-] monkeybutton|4 years ago|reply
[+] [-] mumblemumble|4 years ago|reply
Is the problem that they're not teaching that in college anymore, or is it that the people in charge of these things are non-engineers who don't necessarily even know what a greedy algorithm is, let alone that the principle might apply to business practices and not just software implementations.
[+] [-] marshray|4 years ago|reply
Funny, that's almost exactly how long it takes for the promo video to start auto-playing.
I know for myself I try to skip past titles before getting barraged with the promo.
[+] [-] meowface|4 years ago|reply
>The same tricks that nearly destroyed online journalism now threaten to take over the streaming service.
One of the current front page headlines on slate.com:
>I’ve Been Telling a Lie to Trick Men Into Sex With Me. Is This Really So Bad?
[+] [-] de_keyboard|4 years ago|reply