top | item 7642545

The Case Against ISP Tolls

398 points| acgourley | 12 years ago |blog.netflix.com

254 comments

order
[+] nostromo|12 years ago|reply
Netflix and Google and Amazon and Apple should join forces to let Comcast customers know when they're being throttled.

"Your video is buffering because Comcast is slowing your connection down. Click here to contact Comcast customer support. Click here to find Netflix Certified providers in your area."

Netflix and Google and Amazon and Apple all have great brands with high customer satisfaction. They should utilize all those eyeballs to funnel even more hate toward Comcast.

[+] brandon272|12 years ago|reply
> "Click here to find Netflix Certified providers in your area."

Followed by:

"I'm sorry, there are no Netflix Certified providers in your area."

When Comcast is rated as one of the worst companies for customer satisfaction, customers aren't staying with them because they are ignorant when it comes to the alternatives. They are staying with them because there are no alternatives.

[+] dangrossman|12 years ago|reply
According to Netflix in February, no ISP has ever throttled the service. My understanding is that nobody's intentionally slowed down anyone else, they just failed to allocate additional capital to improve congested parts of their network. Comcast should also still be bound by the "net neutrality" principles they agreed to as a condition of their NBC Universal merger. I'd think if they were singling out any service for throttling, at least one of their 136K employees would blow the whistle to the FCC/DOJ.
[+] xpose2000|12 years ago|reply
I think the real question is... why haven't they already done this? Are they worried about pushback at the political level? Certainly their users would agree.
[+] lern_too_spel|12 years ago|reply
Click somewhere else to cancel Netflix and use Comcast's competing service.

Comcast is holding all the cards.

[+] graylights|12 years ago|reply
Or more directly add a Comcast fee to subscribers on comcast.
[+] zecho|12 years ago|reply
This happens between cable/sat providers and the networks all the time. Company pissing matches in public are terrible for consumers.
[+] whoismua|12 years ago|reply
They have horrible customer service and service for one reason only: because they get away with it. In many markets you have no real choice, and even if you do is the Horrible Comcast or just a as horrible Verizon, AT&T and so on.
[+] notdonspaulding|12 years ago|reply
Comcast can do what it does because Democrats portray the company as a problem with capitalism, while Republicans portray it as a problem with regulation.

In truth, it's both. It's a problem where a company has co-opted well-meaning regulation to snuff out competition, which is a thing capitalist companies do. Meanwhile, citizens point the finger at each other instead of removing Comcast's ability to monopolize.

Either the regulations that make competing as an ISP difficult need to be stripped out (my personal favorite), or the last mile needs to be public property. Either way, Comcast getting to be the only one sitting on that last-mile connection is what has given rise to its stranglehold on the internet.

We need to support fixing this on a local/state level. Support smaller ISPs in their attempts to disrupt the incumbents, and support legislation that actually encourages innovation in the marketplace (even if that means striking policies from the law books).

[+] nostromo|12 years ago|reply
This is the brilliance of what Google is doing. They are making cities compete for Google Fiber. You compete by making it easier, faster, and cheaper to lay fiber.

It seems to be working. Seattle's new mayor recently talked about how the city needs to make it easier to build out internet infrastructure.

http://murray.seattle.gov/murray-seattle-must-be-a-national-...

> Another possible solution includes granting internet companies access to utility poles at little or no charge, so that building more infrastructure is not cost prohibitive.

> We are considering [changing policies] which make it nearly impossible for internet providers to expand existing services without an unusually high super majority of support from neighbors.

[+] maxsilver|12 years ago|reply
As someone who's trying to bootstrap a small, local, last-mile ISP right now, my only wish is that there were many more people that held your opinion on this matter, and were willing to back that opinion with their dollars.
[+] afiler|12 years ago|reply
103 years ago, the UK government nationalized the https://en.wikipedia.org/wiki/National_Telephone_Company. Within two years of that, the US justice department managed to extract from AT&T the https://en.wikipedia.org/wiki/Kingsbury_Commitment. In this, AT&T, unlike NTC, was allowed to keep ownership of its respective telephone network in exchange for interconnecting fairly with its competitors.

There were fights at the municipal level too, but by that point, only a few medium-sized places like Hull in the UK, and Rochester, New York had phone companies that weren't owned by NTC or AT&T.

All of this seems strikingly familiar today, except that there seem to be a lot fewer politicians willing to propose any of the real profound changes like the ones that came a century ago.

[+] runeks|12 years ago|reply
> Either the regulations that make competing as an ISP difficult need to be stripped out [...]

Which regulations are we talking about here? This seems like really low hanging fruit, but of course it probably isn't.

[+] 6cxs2hd6|12 years ago|reply
When you think about it, Comcast NBC Universal is doing the same stuff with Aereo.

Broadcasters get a government monopoly on a finite resource. Broadcasters don't provide a good signal to all customers. Some customers want to use Aereo (that's not the only reason, but a real reason) instead of paying for cable.

But no, the broadcasters want to be paid for not-providing a signal to those customers. Instead of investing to improve their infrastructure, they want to be paid for not improving it. With a straight face.

Of course it's especially ridiculous in the case where it's Comcast NBC, and they are also an ISP. If Aereo wins at SCOTUS, and grows big enough, Comcast NBC can get their pound of flesh as an ISP like they do with Netflix.

It is all well and truly fscked.

[+] lessnonymous|12 years ago|reply
> But no, the broadcasters want to be paid for not-providing a signal to those customers. Instead of investing to improve their infrastructure, they want to be paid for not improving it.

This is the best one paragraph summary of the Aereo case I've heard. Certainly makes it absolutely plain why it needs to be allowed to continue.

[+] fizzbar|12 years ago|reply
This image http://www.willchatham.com/wp-content/uploads/2014/01/net.pn... has been making the rounds on reddit, surprised it hasn't appeared here yet. This is the most visceral/succinct explanation of the consequences of allowing the ISPs to have a "fast lane"...
[+] wmf|12 years ago|reply
The clever thing about Comcast's intentional congestion extortion loophole is that the customer never sees it, so it doesn't look like that image.
[+] anaphor|12 years ago|reply
That picture makes me almost physically ill
[+] higherpurpose|12 years ago|reply
This could be the right time for Netflix to move to P2P, so then most of the bandwidth isn't consumed by them, but by their users.

http://finance.yahoo.com/news/netflix-switched-p2p-video-str...

It's going to take a lot to fight the likes of Comcast in this one, which means Netflix will also need support from the community - so Netflix, please stop it with the boneheaded DRM push on the web. Guess what, if you are going to use P2P, you're going to need a plugin or native app anyway. So just stop trying to corrupt W3C already in a pointless move that can't and won't stop piracy anyway (which I'm sure you already know).

[+] freehunter|12 years ago|reply
Is it Netflix fighting for DRM, or is it the content creators demanding DRM and Netflix simply wanting to move to HTML5 while still being able to continue licensing shows from the creators.
[+] wmf|12 years ago|reply
P2P would shift the congestion from transit links to shared upstream last mile links which will probably make ISPs even more unhappy.
[+] Cakez0r|12 years ago|reply
It would be really interesting if they offered reduced subscription fees for anyone who acts as an edge CDN node for them!
[+] debian3|12 years ago|reply
You only need two or three major website who blackout their services to the Comcast user and redirect them to a page explaining that situation. Imagine if their subscriber can't access to Google and Facebook, and on that redirection page they target ads for alternative provider in your area. Would be nice to see how long Comcast would take to change their mind.
[+] Cakez0r|12 years ago|reply
What about users who have no alternative to Comcast?
[+] kevinchen|12 years ago|reply
That is incredibly user-hostile. Google and Facebook would end up looking like the bad guys, and nothing would change.
[+] jebblue|12 years ago|reply
This:

>> "In this way, Comcast is double dipping by getting both its subscribers and Internet content providers to pay for access to each other."

..and this:

>> "Put simply, there is one and only one way to reach Comcast’s subscribers at the last mile: Comcast."

...should be illegal. This really torques me. This smacks of how AT&T acted in the 80's when I was in my 20's and starting out on my own and how Microsoft acted in the 90's before they got busted, which is what lead me to Linux.

[+] oelmekki|12 years ago|reply
Same problem exists in France with the "Free" (actual business name) ISP.

It's a large ISP here, and it throttles access to youtube (and probably other content providers). It has been so for years. For some reason, there is no public awareness about that, people just say "my internet is not working well".

It's an act of aggression against their customers, to me, since all ISPs are marketing their offers around connection speed. And it's violating net neutrality, too. I don't get how it does not become a major PR problem for ISPs doing that.

[+] liotier|12 years ago|reply
Not just Iliad's Free - I know first hand that others admit internally that they are doing it too... But they choose to let Free be publicly vocal about it while they are just as violent behind closed doors.
[+] andrew93101|12 years ago|reply
Has there been discussion of passing this fee on to customers who access from Comcast in the form of a surcharge? If Netflix hiked prices for customers who were connecting from Comcast networks, it would encourage news of this business practice to spread.
[+] zaroth|12 years ago|reply
Netflix presumably was paying Cogent a lot of money for all those 10Gbit uplinks. Now a few cross-connects and BGP tweaks later, a traceroute shows Netflix packets leaving a Netflix server and going more directly onto Comcast's network. I can only speculate that they turned on some dark fiber to build cross-connects between their cages and Comcast cages. Maybe they were already in the same building...

How much did Netflix's Cogent bill go down? How much of the "work" that Cogent was doing is Netflix now doing themselves? How many hops of the traceroute are owned by Netflix, and how many hops by Comcast?

When I rent a rack in One Wilshire in LA, CA, it comes with power, cooling, and a network link. When I pay let's say $1000/mo for that 1Gbps link, it's not just the link from my rack to their router that I'm paying for. And when I send 1Gbps of packets up through that link, it's not just CoreSite at One Wilshire carrying the cost of delivering those packets.

At some point, yes, I do expect Comcast to get their "fair share" of that $1k/mo. If Comcast is PAID zero dollars for delivering that 1Gbps, then I want this free bandwidth gravy train to propagate all the way back to me. Comcast, please sign me up for 10 of those free 10Gbe links you're giving out, ok? Just tell me where to rent space where I can get those free 10Gbe cross connects to your network.

But then, you can also flip the perspective and ask the question, why doesn't Comcast have to pay for all their bandwidth? Shouldn't they be paying $1,000/mo for their Gbe's?

So why does bandwidth cost money, or more specifically when should bandwidth cost money? Price, as always, is just a means of efficiently allocating scarce resources. Price can only possibly be zero if there is no scarcity. Is it possible there's no scarcity of bandwidth on Comcast's network? Now that would be comical!

I think the answer lies partially in another development. Ask Netflix to chart the cost in terms of Watts and Rack Units to deliver 1Gbps of video streams. My theory would be the cost in terms of space and electricity to generate 10Gbe of video streams has dropped orders of magnitude. This means bandwidth will be an increasing portion of your bill no matter what. It also puts more pressure on scarcity of bandwidth when the ease of which you can generate 10Gbps of useful traffic is down to just a few servers.

[+] maxhou|12 years ago|reply
> My theory would be the cost in terms of space and electricity to generate 10Gbe of video streams has dropped orders of magnitude

Spot on.

You can stream video with basically nginx serving static files, you just need a 10G network card with TSO.

the CPU power needed is an order of magnitude lower than what you need to run full stack server side frameworks.

Just make sure the content fits in your page cache (RAM is cheap), and you will fill the pipe.

In the upcoming month, you will be able to do this with an ARM SOC sitting on a 10cm^2 board, for 5W TDP.

[+] aidenn0|12 years ago|reply
> If comcast is PAID zero dollars...

Comcast gets paid $50/month per subscriber for that bandwidth. It costs Comcast less money to peer with Netflix than to get it from cogent. Comcast wants Netflix to pay to solve a problem that Comcast's customers have.

> Is it possible there's no scarcity of bandwidth on Comcast's network?

Absolutely! In fact, that's part of what the chart shows; when Netflix peered with Comcast, the video quality (which is a proxy for bitrate to the consumer) went up significantly, which indicates that there was no bottleneck on the Comcast network.

> Price ... is just a means of efficiently allocating scarce resources.

Not in the case of a monopoly. Comcast has a monopoly on a large fraction of the broadband market. They also compete directly with Netflix, so they will leverage that monopoly to make it as expensive as possible for Netflix to continue to compete with them. This is Comcast creating an artificial scarcity of paths from Netflix to Comcast's customers.

On another note, I have referred to Comcast subscribers as "customers" above; if agreements like the above continue, then Comcast subscribers are no longer customers, but rather the product that Comcast charges for efficient access to.

[+] sschueller|12 years ago|reply
If Comcast had any competition they would be the ones paying for the better connection to Netflix. It is supposed to be in Comcast's interest to provide their customers with the access they want or the customers would switch to another provider that does offer it. Comcast is selling access to the Internet are they not?

Netflix should be paying Comcast ZERO!

[+] phrasz|12 years ago|reply
They should use classic Cable TV vs. Local Broadcasters tactics: Block all IPs from Comcast and force them to pay access to your content...which happens to be the exact amount they want for the Tolls.

If we consumers have to pay to access to content why shouldn't ISPs?

[+] SourPatch|12 years ago|reply
That might work if Netflix had compelling content. Maybe they do, I don't know. But it is not helping Netflix that Comcast is ranked near the top for ISP market share and also owns a ton of its own content (NBC Universal).
[+] knodi|12 years ago|reply
That would kill Netflix before it hurts Comcast.
[+] dec0dedab0de|12 years ago|reply
This makes me so mad. I kind of wish all of the Big Internet providers like Level3, and Sprint would team up and block all traffic in and out of Comcast. Maybe redirect all http traffic to a page listing alternative ISPs.
[+] johnpowell|12 years ago|reply
There aren't really alternate ISP's. It is Comcast or Centurylink here. And I live in a relatively big city of 200K people.
[+] mikekij|12 years ago|reply
Reading this actually makes me hope that Facebook gets their wifi-beaming balloons up and running quickly. I'd like to support Comcast as little as humanly possible.
[+] wmf|12 years ago|reply
Neither balloons or drones are going to be fast enough to watch Netflix.
[+] 01Michael10|12 years ago|reply
I think it is actually Google looking to deploy Wi-Fi beaming balloons and Facebook is looking at solar powered drones. I vote for Google over drones...
[+] naviwins|12 years ago|reply
I see all these people saying Comcast was feuding with cogent... that is just not the case. For anyone in the hosting business we all know cogent is the cheapest option, and a pretty spotty service as a whole. What do we do when our customers complain and complain? Well, we seek out more reliable options and spend the money needed better serve our customers.

So why does Netflix get to complain? They have already spent the money for the ridiculously one sided peering(Netflix sending tons of traffic and Comcast sending close to none). You would think Netflix would be happy to spend this money to better their service! Right? Or is it so detrimental to their cash flow that they would rather complain? They should have just told their customers "hey, we heard you were having some problems with our service, look what we did for you!"

In the end this is just a smear campaign to distract Netflix customers from the price hike that is coming. It is obvious Netflix was going to raise its subscription prices eventually. So why not point the blame to a company who is already disliked. The majority of HN seems to be pretty smart people, it would be a shame if you fell for this rather underhanded ploy.

[+] josephlord|12 years ago|reply
Who sends traffic is just easier to measure, the person receiving it may want it more and in the Netflix case they are requesting the traffic.

In any case all Comcast's traffic is for their customers, when peering needs to be balanced to be free is when to parties are offering transit to other networks beyond themselves.

[+] nkcmr|12 years ago|reply
I really hope Netflix has what it takes to do what is necessary if it comes to that point. And what it is going to take is for Netflix to simply blackout Comcast.

But actions like that can be easily spun out in the media (media of which has very close ties to Comcast) to make Netflix look bad.

What would make it better is for all of the major players (Google, Facebook, Twitter, Apple) to collectively blackout Comcast, whilst educating the public on how they are being used and screwed at the same time.

[+] josephlord|12 years ago|reply
If Netflix are right an Comcast are providing insufficient connectivity through to all of the major transit networks (rather than just Cogent) it feels to me as if Comcast are not providing the internet connection speeds that they are selling. It would be interesting to see what would happen if Netflix bought from all the many transit providers and spread the load across them, would all of Comcasts general service grind to a halt.
[+] brianmcdonough|12 years ago|reply
The problem is sufficiently complex and boring to favor Comcast. What I would like to know is this: It is already a regional monopoly in high speed internet because no on can touch their speeds, why would anyone consider extending their monopoly nation-wide?
[+] clubhi|12 years ago|reply
I dropped Netflix about a month ago when I first heard they made this agreement with Comcast. They are part of the problem.
[+] pixelcort|12 years ago|reply
I think the real problem is unlimited internet connections.

If ISP customers paid per-GB and/or per-Mbps, then the ISP shouldn't care if it's Netflix or Hulu or YouTube being consumed.

The issue is that today, it's difficult to get a good feel for how much bandwidth is being used at any given moment. Unlike a water faucet, our devices don't really have a knob we can adjust the speed with.