top | item 7872168

Netflix responds to Verizon

508 points| chrisacky | 11 years ago |scribd.com

258 comments

order
[+] dctoedt|11 years ago|reply
Nice jab with (what seems to be) an allusion to the New Jersey bridge-closing scandal that engulfed Gov. Chris Christie: "... like blaming drivers on a bridge for traffic jams when you're the one who decided to leave three lanes closed during rush hour."
[+] ColinDabritz|11 years ago|reply
I hope Netflix makes this a standard feature for all ISPs. Bonus point for more transparency, e.g. network graphs comparing peak to off-peak.

This sort of visibility into the real problems means the ISPs can't hide behind their lies.

[+] Bud|11 years ago|reply
You're very optimistic.

Myself, I think it's far more likely that ISPs are about to successfully run out the clock on all this, kill net neutrality, and then they won't have to ever worry about this or any "transparency" efforts again. Everyone can just pay more for ever-crappier service, and there will be no relevant data to examine, because there will be no "standard" level of service anyone has access to, and therefore no way to compare one level of service to any other level of service at another time or via someone else's product...or at least, no way to do anything about it other than impotently gripe.

[+] oostevo|11 years ago|reply
It sounds like they might:

"We are testing this type of messaging across the U.S. with multiple providers" [end of 1st paragraph]

and

"The current transparency test ... is scheduled to end June 16 and we are evaluating rolling it out more broadly. Regardless of this specific test, we will continue to work on ways to communicate network conditions to our customers." [last paragraph]

[+] deveac|11 years ago|reply
|This sort of visibility into the real problems means the ISPs can't hide behind their lies.|

Unfortunately, all the transparency in the world can't effect change when the consumer lacks choice in the marketplace.

ISPs need to be classified as common carriers.

[+] jobu|11 years ago|reply
What is the messaging they're displaying to Verizon customers?
[+] jsz0|11 years ago|reply
> ISPs can't hide behind their lies.

NetFlix isn't exactly being honest here either. They could easily delay the start of video playback to allow for more caching on congested networks to offer their customers a better experience. They could also allow customers to queue videos to be downloaded/cached locally in advance. These things would provide a better experience to NetFlix's customers so why aren't they doing them? I suppose they cost too much to develop or would increase NetFlix's costs with content providers. In other words NetFlix is trying to maximize their own profits and throwing a fit that ISPs are doing the exact same thing.

[+] Osiris|11 years ago|reply
I believe the core problem here is that ISPs and infrastructure are owned by the same companies. Based on evidence from other countries, like Japan, these problems would largely go away if it was illegal to own both the infrastructure and provide service to customers.

Is it possible that anti-trust laws could be used to force these large ISPs to break up into separate companies, one that owns and provides the network and the other that only leases the lines and provides service?

Such a breakup would allow new, smaller, ISPs to leverage the infrastructure while provided superior service and potentially better pricing models.

Of course there would still be the problem of competing infrastructure companies. Some of that could be alleviated by increasing competition through public/private fiber installation projects.

[+] seynb|11 years ago|reply
>I believe the core problem here is that ISPs and infrastructure are owned by the same companies.

Exactly. MSO (Cable) & Telco ISPs that offer video products are already prioritizing their own VoD (Video-on-Demand) offerings over data to ensure QoS (quality of service). Whether by over-provisioning the virtual circuits to their customers, not amortizing the data consumed against a hard or soft cap limit, or directly peering the VoD servers with the edge routers, ISP video gets the priority because that is where the big profits are. There is no value proposition for BIG ISP to provide free peering with their competition.

[+] Silhouette|11 years ago|reply
I believe the core problem here is that ISPs and infrastructure are owned by the same companies. Based on evidence from other countries, like Japan, these problems would largely go away if it was illegal to own both the infrastructure and provide service to customers.

That is certainly an idea worth exploring, but I'm not sure it's an automatic win.

We could consider European railway networks as a somewhat analogous situation. In several countries, one business or public entity owns and maintains the tracks and related infrastructure, while one or more others own and operate the trains themselves.

We have a better service here in the UK today with that model than we used to with the nationalised all-in British Rail, in some respects at least. On the other hand, the costs of that system are crazy, both in ticket prices and government subsidy, so it's far from clear whether the improvements have come because of the split in responsibilities or despite it.

We're not the only ones whose system doesn't always work, though. Every now and then, you see a real howler due to miscommunication, as IIRC the French had recently when it turned out that the super new trains wouldn't be able to fit next to the platforms at a load of older stations.

It seems to me that the main problem with ISPs in the US is not so much being both infrastructure provider and service provider as it is having little if any real competition on either count. Rather like the banking system, the US appears to be maybe a decade behind most of the developed world in this respect, for no particularly good reason. I can only assume it's a consequence of politics and/or questionable business environment.

[+] manicdee|11 years ago|reply
Just make sure you do it properly the first time, and don't trust the telcos when they tell you they'll "self regulate". Otherwise you'll end up with the mess that Australia has, which led to the previous Government setting up the NBN project.

Not only that, but any time a retail ISP submits complaints about a customer's service, Telstra calls the customer directly and tells them that their problems will go away if they subscribe to a Telstra plan (and the problems do go away, strangely enough).

So make sure there is a proper separation of the infrastructure and retail portions of the telco behemoth right from day 1. Otherwise you'll be dealing with a situation that is worse than what you have now.

[+] danielweber|11 years ago|reply
So the DSL infrastructure?
[+] higherpurpose|11 years ago|reply
I hope Netflix doesn't give in, and calls Verizon's bluff. It's highly unlikely Verizon will sue Netflix, and even if it does just because they are so angry with Netflix over it, it will most likely come out in favor of Netflix, once the Court orders Verizon to show what's really happening behind the scenes and who's fault really is.

In fact, from what I've noticed, Verizon has already lowered their tone about this, and is backing away from threatening Netflix with the lawsuit. So carry on!

These ISP's promised good service to their paying customers, regardless of the conditions. It's their responsibility to live up to those expectations.

[+] mcintyre1994|11 years ago|reply
Someone on Reddit pointed out that it would be a huge PR win for Netflix if Verizon did sue and they got the right documents in discovery. Depending on the scope maybe they could possibly get things as wide ranging as actual customer speeds as well as the peering points/capacity etc.
[+] jusben1369|11 years ago|reply
Written to Verizon but written for everyone but Verizon. This is a PR battle started by Verizon but I think Netflix just went up 3 - 0.
[+] Russell91|11 years ago|reply
Netflix lands more squarely on the customer's side on this topic. They have everything to gain from PR coverage and Verizon everything to lose. Seems like a bad move to me that Verizon decided to boost the awareness of this issue in the first place. Perhaps that wasn't the intention, but it was the effect.
[+] xster|11 years ago|reply
ya, was just thinking how non-lawyer-cat-speak the letter is
[+] taylorwc|11 years ago|reply
Ugh. Good for Netflix. Any consumer who doesn't have a strong opinion in favor of net neutrality should read this sort of thing. It's hard to think of a relationship I detest more than my relationship with my ISP/MSO.
[+] markbnj|11 years ago|reply
Yes, indeed, good for them. The ISPs are counting on general ignorance and a PR battle to win them the right to charge incoming tolls on their networks. This has to be resisted. However, I read this morning that Netflix caved and agreed to remove the messaging early. I hope it's not true, and that they keep up the fight. Three Netflix accounts in our house.
[+] gnu8|11 years ago|reply
How about a link to the document instead of some fly-by-night web host with a cheesy flash PDF viewer?
[+] dpcx|11 years ago|reply
Scribd isn't exactly a "fly-by-night web host". They've been around for almost 10 years already.
[+] jboons|11 years ago|reply
I had a pretty easy time reading it. Was it difficult for you?
[+] ethnt|11 years ago|reply
There's a large orange "Download" button directly to the right of the document.
[+] dmethvin|11 years ago|reply
They are saying "The Verizon network is crowded right now" but in the letter they say that the problem is with "interconnection congestion". So yes the problem lies with Verizon and its hesitancy to open up peering points, but if I were a Netflix lawyer that wording would have me concerned since Verizon's network is not crowded. It's like having a crowd outside a bar waiting to get in.
[+] klodolph|11 years ago|reply
It's like having a crowd inside a bar waiting to get out.
[+] Angostura|11 years ago|reply
The congested interconnection point is part of Verizon's network, surely.
[+] AnthonyMouse|11 years ago|reply
Netflix should beg Verizon to sue them for false advertising. You can imagine the PR coup when Netflix comes out victorious.
[+] brandoncordell|11 years ago|reply
Isn't that essentially saying there's a crowd outside the bar waiting to get in because the bar itself is already packed?
[+] chernevik|11 years ago|reply
Kudos to Netflix, this is great strategy. But I wish the letter were a touch better written. They might attach a written-for-laymen description of the interconnection problem (so they can keep the response itself punchy).

As I understand the problem, the congestion builds at the interconnection because Verizon's routers at that point at working at capacity. Verizon could solve the problem by adding routers at that point. I honestly don't understand what Verizon wants Netflix to do, unless it is to extend the Netflix data provision deeper into the Verizon network -- essentially adding routers at the point where Verizon is supposed to be maintaining routers. (I wonder if that is what this Open Connect program is about?)

If that's right, I wish Netflix would provide some further detail explaining the problem. Otherwise they risk having readers -- and politicians -- go into glazed eyes and presume that this is just some inscrutable battle among corporate giants.

[+] hga|11 years ago|reply
"I honestly don't understand what Verizon wants Netflix to do...."

Die in a ditch.

Seriously. Per the last big discussion on this, from https://news.ycombinator.com/item?id=7858919 from a Verizon press release, "Verizon had a total of 5.8 million FiOS Internet and 5.0 million FiOS Video connections at the end of the [2nd] quarter [of 2013]". I would infer from that press release that they have 3.1 million plain DSL customers.

Netflix et. al. are an existential threat to their FiOS Video business, especially since to my knowledge Verizon is the only holdout in putting caps on landline Internet connections (e.g. AT&T is in much less danger with 150-250 GiB/month, $10/for each additional 50 GiB).

So I'm assuming there's nothing Netflix can do to satisfy Verizon besides giving up, going out of business, etc. Which, with cap and congestion ISP policies, is not beyond the realm of possibility; since the DVD renting business went sour, I've long thought for this reason Netflix was a high risk venture.

[+] lstamour|11 years ago|reply
For your second paragraph, that is what Open Connect is about -- put netflix hardware inside the ISP's own network to reduce congestion at interconnects.

That said, it's also an inscrutable battle among corporate giants, but one that people care about because in general, most people hate their ISP but like internet companies like Netflix and so put up with things.

For up-to-date analysis on Internet issues, read up on dslreports.com as each stunt happens. It's not enough to hit mainstream news often, but if you follow it online you'll pick up on patterns. Here they cover this PDF (the last paragraph links to backstory): http://www.dslreports.com/shownews/Netflix-Wont-Back-Down-on...

[+] wmf|11 years ago|reply
Netflix paid Verizon to upgrade their network, but Verizon hasn't finished the upgrades because it takes them months to order and install routers. Netflix knows that these upgrades are coming but is still publicly shaming Verizon, perhaps trying to encourage them to work faster. Presumably Verizon would prefer Netflix to be patient.
[+] crazy1van|11 years ago|reply
It is so aggravating to have Verizon pitch an upgrade to FiOS Quantum(!) every time I turn my DVR on. If my 25mbps connection can't stream a 5mbps video why would I pay extra for 50mbps?

No thank you, Verizon.

[+] uptownhr|11 years ago|reply
I'm thinking from a Consumer's stand point right now. No matter how I see it, it looks like ISPs are at fault in this matter but I've been questioning what the outcome will be for the Consumers in the end.

If Netflix doesn't pay, wouldn't Verizon and other service providers transfer the cost to Consumers? Or will this also be not allowed for ISP to do? If ISPs want to make more money and charge more, why would they not transfer the cost to Consumers if they cannot charge Netflix and the likes.

[+] gdulli|11 years ago|reply
Am I the only one who doesn't use streaming video and doesn't want my ISP rates to go up because a vast majority of the traffic they now have to deal with is video and their architecture needs to be built out in a way that it otherwise wouldn't?
[+] danudey|11 years ago|reply
The problem, as I see it, is that Verizon/Comcast (and other ISPs) are overselling, and the amount of traffic that Netflix is generating is showing that. Verizon sells X customers Y-bandwidth connections, but they don't have X*Y worth of uplinks, so they can't supply data to the people who want it.

Until now, this hasn't been a problem, because most people didn't max out their connections, and if they did they didn't do it for very long. The very nature of Netflix's business, though, is to serve the best traffic that the customer's connection can handle, and that starts to add up. Unfortunately, it adds up so much that in some cases (e.g. Comcast) you can't even serve VHS-quality content to all the customers who want it.

Because Verizon is overselling their bandwidth, they're now having issues where they can't service the demand that they sold their customers on. Instead of upgrading their upstream connections (which would cost them money, which they have a lot of), they want Netflix to pay for it. The logic is that Verizon wouldn't have these problems if Netflix weren't sending so much content; that argument breaks down when you consider that Verizon sold their services based on available bandwidth and now can't provide it; Netflix is just the service that exposed the problem.

Netflix has other solutions, though; they can provide servers that get hosted in the ISP's network to provide content to users without traffic coming from outside the ISPs' networks (thus freeing up their normal uplinks for regular traffic), or by peering with Netflix at various points (which accomplishes the same thing).

Unfortunately, neither of these solutions involves Netflix giving money to Verizon, which is why Verizon doesn't want to bother with going to the time and trouble when they can force Netflix to pay for their infrastructure upgrades instead.

[+] danielweber|11 years ago|reply
I appreciate your POV that what you have now is "good enough" and it doesn't feel right to be taken along to support other people's habits. (And it's totally rotten you got downvoted.)

However, ISPs are a shared resource. Just like I tell people who act shocked when their speeds aren't guaranteed, sometimes the needs of the shared network override yours. You are getting a tremendous discount over a private line by bundling your whims with a bunch of your neighbors.

That said, if your current service is "good enough," you might try using wireless or DSL.

[+] Iftheshoefits|11 years ago|reply
Probably you aren't the only one, but then again I don't, either.

It is a possibility (likelihood, even) that Verizon and other providers will raise rates, even though they aren't delivering on their commitments at current rates. I frankly consider residential ISP service providers who do what Verizon does to be in breach of the spirit of their contracts, even if the "best effort" language offers weasel-word loopholes out of being in breach (legal) of their contracts. I don't think they should be allowed to raise their rates until they demonstrate they can meet their current commitments under their current rate structures. But there is an infinitesimally small chance of regulatory, legislative, or court actions that will enforce that, especially in this country today, so I'm basically just waiting for the boost in monthly cost for my internet service.

[+] wpietri|11 years ago|reply
Yes, you are the only one, or close enough that it makes little difference.

Back in Ye Olden Dayes, when techies were the main audience for technical products, we could reasonably expect offerings that catered to our needs. But for quite some time, the industry has been driven by mainstream consumers. E.g., the last smartphone built for a nerd audience was the Palm Treo. The current ones are all consumer focused.

We're along for the ride now, and video in particular has been a major driver for bandwidth demand.

It could be that there will be enough we-don't-need-much-bandwidth consumers that somebody will offer a not-good-enough-to-stream package that will be a better deal for you. But again, that's a consumer pricing option.

[+] droopybuns|11 years ago|reply
When video is consuming most of the internet transport links, it only makes sense that website operators should also shoulder rate hikes. /s

We who understand how the network works and what is actually happening to it and simultaneously observing the net-neutrality screaming of most people should take note. People do not behave rationally, nor do they care about anything nearly as much as their internet cat videos or "house of cards" episodes.

[+] nhangen|11 years ago|reply
Is that real? It doesn't read like it was written by an attorney, and there was even a grammar mistake in the first paragraph.
[+] panabee|11 years ago|reply
hopefully netflix continues this push toward broadband transparency. as justice louis brandeis said, "sunlight is said to be the best of disinfectants." it seems unreasonable for verizon and broadband providers to advertise, and charge consumers for, high-quality networks -- then try to pry money from data suppliers like netflix when usage rises. that said, is there a fair counterargument that justifies verizon's desire to charge both consumers for high-speed access and netflix for increased network usage?
[+] cipherzero|11 years ago|reply
What's the Open Connect program they mention? how does it work?
[+] tieTYT|11 years ago|reply
I don't know how Netflix figures out that it's Verizon that's slow, but lets say it's with some software that anyone could use. Wouldn't it benefit the entire Internet to open source that software so that anyone could use it? Or, barring that, provide an API that anyone can query.

This way, any company can use the software and report the same thing to their customers. Lower the barrier to entry for Amazon/Google/Hulu/your startup to join the fight and inform the people.

[+] bryondowd|11 years ago|reply
Maybe I'm nitpicking, but in their bridge analogy, wouldn't the drivers be more analogous to the users than to Netflix? If anything, Netflix would be the city full of employment opportunities that drivers are commuting to and from across the bridge. So a more fitting comparison would be 'like blaming the city at the other end of the bridge for traffic jams..'

Either way, brilliant response, and hopefully Verizon gets the hint.

[+] thatthatis|11 years ago|reply
They might as well have addressed that letter to hacker news with a cc to verizon. Well executed PR move.
[+] oldmanjay|11 years ago|reply
What I find most interesting about these topics is how many people come out to contribute to them. Nothing gets people riled up like vaguely threatening the free flow of entertainment.

I wish I know how to direct this energy into something a little less disappointing.

[+] asc123|11 years ago|reply
Netflix should just stop serving to Verizon and Comcast. the outlash would force the FCC and companies to recognize.