"On November 22, after being informed by Comcast that its demand for payment was 'take it or leave it,' Level 3 agreed to the terms, under protest, in order to ensure customers did not experience any disruptions."
Leave it. Level3 set a terrible precedent here by paying the shakedown money. Even with their subsequent complaints, they've now put Comcast in a much better position by implicitly acknowledging (whether true or not) that Comcast brings more value to the table by providing access to its subscriber base than content-providers bring to the table by supplying content to subscribers.
Like the old Churchill story goes, "now we're just negotiating the price."
(See PS #4 below for a more benign interpretation.)
--
PS #1: Yes, of course it might have been expensive for them to play hardball here, with respect to the short-term harm it might cause their customers. That's exactly the point. By acknowledging that the harm to Comcast would be lower (which I actually doubt), they've significantly strengthened Comcast's position against other carriers and perhaps even with regulators.
PS #2: Game theory would suggest that it's critical for Netflix to repel this assault now, even at the cost of short term pain. Once this precedence is established, Comcast will be able to extract nearly all of the profit from Netflix sales to Comcast customers. Each time Comcast raises the toll, it would be in Netflix's rational self interest to go along so long as they are still making some net profit on the channel (modulo opportunity costs and whatnot).
PS #3: It's not uncommon for carriers to play hardball. The big Tier 1's have been trying to prevent the ascension of Cogent into their club for years, which (along with Cogent's aggressiveness) resulted in some nasty peering dispute stalemates in 2005 (with Level3) and 2008 (with Sprint), each time "breaking the internet."
PS #4: It's also possible that Level3 is representing this as a net neutrality issue when in reality it's just a peering dispute. Traditionally local ISPs have had to pay for transit across the networks of the big carriers. When a local ISP becomes a large regional or national one (Comcast), they have enough leverage to extract no-charge ("settlement-free") peering agreements with most of the major carriers. This is probably the first case, though, of a cable-company ISP becoming large enough to extract transit fees from a big international carrier. That's noteworthy, but if it is content-neutral, it's still the way the internet has always worked.
I don't think I've ever seen anyone cherry-pick various points from other posts without attribution before. Anything wrong with responding in the the thread instead of constantly updating your post with other people's (including mine) comments?
Editing a post is typically for the purpose of fixing typos, making your (own) thoughts more concise or clear.
It's not for the wholesale amalgamation of other people's posts, as useful a function as you think you might be serving.
Let your original post stand on its own merit, or don't post it.
On the peering front, interesting case study is MWeb, an ISP in South Africa. They have the opposite problem in that they recently refused to continue to pay other ISP's in the country for the right to peer with them. So all local traffic to and from their network is routed internationally since they have their own international links.
Framing the argument as a peering dispute is an interesting strategy. But I think the differentiator is that if peering is not allowed, there is no alternative route to the customer. Whereas if it were a traditional peering dispute, the quality of the service would degrade with longer routes rather than be terminated altogether.
"“On November 19, 2010, Comcast informed Level 3 that, for the first time, it will demand a recurring fee from Level 3 to transmit Internet online movies and other content to Comcast’s customers who request such content."
...and other content.
I'd like to know the entire story here. Is Comcast demanding that Level3 Pay a fee to deliver online movies, or is Comcast demanding peering fees. One of those is content neutral, the other is not.
Note - as I read it, Comcast is demanding peering fees from Level 3. Big Whoop.
I'm fine with Comcast demanding money from inbound-feeds to transit their network, and shutting them off if they don't pay it. That's why any sane content provider multi-homes . Think about the position Netflix put L3 in by announcing an "exclusive" CDN deal with L3. Comcasts negotiating position with L3 got a whole lot better.
This is not unusual on the internet - think about how often cogent gets de-peered because it's unwilling to pay peering fees. I've had a ton of colleagues get blackholed because they made the mistake of single-homing through cogent. Ironically, L3 depeered from Cogent a few years ago... So, what's good for the goose...
Net-Net: Always Multi-home through at least a couple Tier 1 providers, so when one of them gets depeered, your customers can still communicate with you. Be wary of exclusive transit deals.
Anybody with some actual insight care to comment on what's really going on here?
While peering disputes have happened throughout Internet history, it is unusual to see an eyeball ISP trying to charge a tier 1. Usually the money flows the opposite direction.
I think your comment is right on the money. The devil is undoubtedly in the details here. We know netflix just switched away from akamai for video cdn to l3. Aside from scale, it's easy to guess one of the major reasons was price - akamai is one of the highest cost providers and L3 has been aggressively bidding on work. We also know that akamai has way more POPs or at least caching locations (2000+ is the pr) and is very well positioned to locally deliver on comcast. L3 video CDN in contrast is more in the dozen or two POP range, and several of those may not yet be up to the weight of netflix traffic. If comcast is going to have to handle a ton more national traffic themselves because netflix is saving money by switching to L3 it only makes sense for comcast to try to offset those costs by charging for transit.
What if I started CheapoStreamCo and started undercutting everyone in the business, but the only place I was on net was NYIIX. Think I'd be getting settlement free deals? Hell no.
If we can set aside the "Comcast is inherently evil" mindset for a moment, this to me reads simply that Comcast notified Level 3 that they were terminating a previously agreed upon settlement-free peering [1] agreement where Comcast and L3 find exchanging traffic without charging each other mutually beneficial.
This kind of thing is not an unusual event in the history of the Internet and can happen for any number of reasons -- including arguably justified criteria such as unbalanced traffic ratios, as well as business strategy (or greed, depending on which side you are on).
Level 3 has elected to frame this as a net neutrality issue and take the dispute to the press and the court of public opinion, but it's probably about a different question: who will pay, and how much will each party pay, for the infrastructure required to deliver content from L3's customers to Comcast's customers?
If viewed in this light, this peering agreement renegotiation is Comcast pushing back some of the costs on L3 and/or it's customers (i.e. streaming video providers). The other alternative would be to raise it's own customer rates.
It's possible that Comcast is being greedy and leveraging their position to rent-seek, but it's also possible that the traffic mix that resulted in the previous peering agreement is now significantly out of whack, justifying this renegotiation. It's definitely not black and white, and this is only one side of the story.
Pretty damn good framing if that's what it is. However, if they really are charging L3 just to deliver competitor's video content online that's pretty much pisses everyone off. I write this using Google's DNS because last night comcast's default DNS was broken. What a terrible company that only stays in business because of near monopoly powers :/
I'm sure it had nothing to do with this announcement from two weeks ago:
"Netflix (NASDAQ:NFLX) announced Thursday (Nov 11th) that it has signed a multi-year deal with Level 3 (NASDAQ: LVLT) for Level 3 to become a primary content delivery network (CDN) provider for the online movie rental company."
This is why we need to use encryption for everything, not just shopping, SSH, and banking. The less distinguishable packets are from noise, the more open the Internet can be.
Sadly, I'm beginning to envision the future of the Internet as a bunch of point-to-point VPN connections to trusted sites, almost exactly like UUCP or the BBSes of days past. VPNs are already essential if you live in certain countries (China, the US if COICA had passed, etc.) or want to BitTorrent your TV shows without getting sued for 30 million dollars. Someday, it might be required just to watch Netflix or search with Google. Sad.
In what world do 65536 octets representing a video cost more to deliver than 65536 representing an email? A world where greedy ISPs can tell the difference.
The ISP can still tell where the data is coming from/to and how much of it there is. Encryption is not a solution to the threats against net neutrality.
It's not clear that encryption would make any difference in this case, because presumably Comcast can easily measure the traffic going through their direct links to Level 3.
You're doing that whole "charge by cost" thing that everyone here seems to frown upon, preferring instead to recommend "charge by value".
In what world can you extract more cash from 65536 octets representing a video than 65536 octets representing an email? A world where end users value them differently.
"On November 22, after being informed by Comcast that its demand for payment was ‘take it or leave it,’ Level 3 agreed to the terms, under protest, in order to ensure customers did not experience any disruptions"
I do not understand how Comcast had leverage in that situation. Had Level 3 told them no then Comcast customers would have been affected. Why is Comcast even negotiating with Level 3 and not the content providers?
"Customers" in this situation mean the big streaming players like Netflix and Hulu. Comcast would surely point fingers at the streaming services (Netflix, Hulu, etc) who would then have to move off Level3. A pretty ballsy move and exactly why Net Neutrality is important.
I think they went for Level3 because average users have no idea who they are. Attacking Netflix gets you in the NY Times. Attacking Level3 gets you on a BusinessWire press release.
Any content provider who's single-homed to Level 3 (which may include L3 CDN customers like Netflix) would become unavailable to Comcast customers if they disconnected.
"Comcast on Monday rebuffed the notion that the new fees were related to Netflix by saying that the type of traffic distributed by Level 3 was irrelevant. Joe Waz, a senior vice president at Comcast, says it has had a peering agreement with Level 3 to swap traffic fairly evenly. Now Level 3 is sharply increasing its traffic, he said, while resisting a commercial agreement to pay for that.
Comcast is “already carrying huge amounts of video to our high-speed Internet customers every day through commercial arrangements, and it seems to be working for everybody else,” Mr. Waz said. “Level 3 is trying to change the rules of the game.”"
This makes the most sense, especially considering Level 3's very careful wording in their press release.
This kind of thing happens all the time, relatively speaking. There's really nothing to care about here.
If you get upset over this, you're just playing into Level 3's hand. They're exploiting the "net neutrality" concept to try to get a better deal out of Comcast, even when this has nothing to do with it.
I'm glad this is happening now. The concept of Net Neutrality has been too abstract for most consumers and legislators thus far, so it's been easy for telcom lobbyists to manipulate the narrative.
Something concrete and egregious needed to happen before the average consumer would understand what it really means to allow companies to filter the type of traffic that people are allowed to get through their pipes.
Comcast is going about it in a way that consumers won't understand. I've seen a lot of people expecting things like tiered pricing (think the "YouTube package"), but this is much more dangerous.
Comcast is going to isolate the content upstream so that consumers won't ever know that there was other content accessible, just like they don't know about the TV channels that Comcast doesn't carry.
If we had proper amounts of competition in ISPs the entire net neutrality problem would simply disappear. It's only because we're forced to negotiate with monopolies that net neutrality is an issue at all.
If I was Level 3 I would have said "leave it" to Comcast and let Comcast block the service for all their customers. It would have caused a bigger media sensation and outrage by Comcast customers against their ISP. By giving in Level 3 has just encouraged this "tollbooth" attitude that ISPs are developing.
Netflix's software traffic balances among Akamai, Limelight, and Level(3).
If Comcast blocked Level(3)'s Netflix hosting, customers would still get streams, just pulled from Akamai and Limelight.
I'd guess that Level3 is counting on "Tier 1 backbone" peering agreements to make delivering this bandwidth affordable while avoiding having to negotiate with each cable co separately, while Akamai and Limelight have direct agreements with the cable companies already.
If this is what's going on, that would give Comcast leverage.
If enough Netflix customers live in Comcast-only areas, Netflix would have had little choice but to lean on Level 3 to capitulate. Even if the customers are willing to switch ISPs just to get Netflix, they wouldn't have had the option. That translates to less money for Netflix, more pissed off customers for Comcast, and a true win for nobody. With Netflix and Level 3 falling on their swords, their customers at least come out ahead in the short term.
In the long term, it's obviously disastrous. At some point, companies will have to refuse en masse to pay the protection money. In the areas where it has a monopoly, Comcast could stomach losing Netflix, but not another 10 or 20 big sites on top of it. There are antitrust issues with collective refusal to pay, but there's got to be a way around that.
It seems to me that Netflix management is a little more forward-thinking than that. Hastings, et al. know that streaming is going to be their primary market and if they're having to pay off every possible intermediary in the future, their margins are going to disappear completely.
While certainly not an ideal situation, I could easily see Netflix going after Comcast with highly-visible notices to their Comcast customers rather than pay them off and set a very, very bad precedent.
Ideal solution would be for Netflix to offer a discount to users on ISPs that don't do this. Importantly, send an email with "this is what you would pay on the different ISPs serving your area".
How can Comcast argue they incur so much costs? when other providers are able to offer triple-play (isp/cable/phone) offerings for a third of the price or less. Example: in France Proxad (http://www.free.fr) offers 179 cable channels, on demand movies and shows, 18 Mbits DSL, free unlimited VOIP calls to the whole country + 40 foreign countries - all for $40 flat (no catch - I have used them happily for 5 years and I am still a customer in France where I own a home).
Comparing France to the US is nonsensical. US is obviously way bigger and thus the network infrastructure required is much more complex, which greatly increases the cost to provide connection to each household.
Comcast pulled quite a bluff, but I don't see how they can win in the long term. As a Comcast subscriber, I'd more than happy to switch to a different ISP if Comcast starts to limit access to Netflix or any other content provider.
As a Comcast subscriber, I'd more than happy to switch to a different ISP if Comcast starts to limit access to Netflix or any other content provider.
How many broadband ISPs are there in your area? In many areas, there are no more than two. If they both adopt this policy, without net neutrality you're out of luck.
As Comcast subscriber, I'm more than happy to switch the minute I have a viable alternative. Whether its a constant barrage of billing issues, poor connections, terrible customer service, abusive monopoly actions, the list goes on...
[+] [-] tc|15 years ago|reply
Leave it. Level3 set a terrible precedent here by paying the shakedown money. Even with their subsequent complaints, they've now put Comcast in a much better position by implicitly acknowledging (whether true or not) that Comcast brings more value to the table by providing access to its subscriber base than content-providers bring to the table by supplying content to subscribers.
Like the old Churchill story goes, "now we're just negotiating the price."
(See PS #4 below for a more benign interpretation.)
--
PS #1: Yes, of course it might have been expensive for them to play hardball here, with respect to the short-term harm it might cause their customers. That's exactly the point. By acknowledging that the harm to Comcast would be lower (which I actually doubt), they've significantly strengthened Comcast's position against other carriers and perhaps even with regulators.
PS #2: Game theory would suggest that it's critical for Netflix to repel this assault now, even at the cost of short term pain. Once this precedence is established, Comcast will be able to extract nearly all of the profit from Netflix sales to Comcast customers. Each time Comcast raises the toll, it would be in Netflix's rational self interest to go along so long as they are still making some net profit on the channel (modulo opportunity costs and whatnot).
PS #3: It's not uncommon for carriers to play hardball. The big Tier 1's have been trying to prevent the ascension of Cogent into their club for years, which (along with Cogent's aggressiveness) resulted in some nasty peering dispute stalemates in 2005 (with Level3) and 2008 (with Sprint), each time "breaking the internet."
PS #4: It's also possible that Level3 is representing this as a net neutrality issue when in reality it's just a peering dispute. Traditionally local ISPs have had to pay for transit across the networks of the big carriers. When a local ISP becomes a large regional or national one (Comcast), they have enough leverage to extract no-charge ("settlement-free") peering agreements with most of the major carriers. This is probably the first case, though, of a cable-company ISP becoming large enough to extract transit fees from a big international carrier. That's noteworthy, but if it is content-neutral, it's still the way the internet has always worked.
[+] [-] ghshephard|15 years ago|reply
Editing a post is typically for the purpose of fixing typos, making your (own) thoughts more concise or clear.
It's not for the wholesale amalgamation of other people's posts, as useful a function as you think you might be serving.
Let your original post stand on its own merit, or don't post it.
[+] [-] mwcremer|15 years ago|reply
[+] [-] mmaunder|15 years ago|reply
http://bit.ly/b8Bslu
Framing the argument as a peering dispute is an interesting strategy. But I think the differentiator is that if peering is not allowed, there is no alternative route to the customer. Whereas if it were a traditional peering dispute, the quality of the service would degrade with longer routes rather than be terminated altogether.
[+] [-] ghshephard|15 years ago|reply
"“On November 19, 2010, Comcast informed Level 3 that, for the first time, it will demand a recurring fee from Level 3 to transmit Internet online movies and other content to Comcast’s customers who request such content."
...and other content.
I'd like to know the entire story here. Is Comcast demanding that Level3 Pay a fee to deliver online movies, or is Comcast demanding peering fees. One of those is content neutral, the other is not.
Note - as I read it, Comcast is demanding peering fees from Level 3. Big Whoop.
I'm fine with Comcast demanding money from inbound-feeds to transit their network, and shutting them off if they don't pay it. That's why any sane content provider multi-homes . Think about the position Netflix put L3 in by announcing an "exclusive" CDN deal with L3. Comcasts negotiating position with L3 got a whole lot better.
This is not unusual on the internet - think about how often cogent gets de-peered because it's unwilling to pay peering fees. I've had a ton of colleagues get blackholed because they made the mistake of single-homing through cogent. Ironically, L3 depeered from Cogent a few years ago... So, what's good for the goose...
For a taste of how this type of negotiating constantly is going on in the background, see: http://www.renesys.com/tech/presentations/pdf/nanog43-peerin...
Net-Net: Always Multi-home through at least a couple Tier 1 providers, so when one of them gets depeered, your customers can still communicate with you. Be wary of exclusive transit deals.
Anybody with some actual insight care to comment on what's really going on here?
[+] [-] wmf|15 years ago|reply
[+] [-] trotsky|15 years ago|reply
What if I started CheapoStreamCo and started undercutting everyone in the business, but the only place I was on net was NYIIX. Think I'd be getting settlement free deals? Hell no.
[+] [-] woodrow|15 years ago|reply
This kind of thing is not an unusual event in the history of the Internet and can happen for any number of reasons -- including arguably justified criteria such as unbalanced traffic ratios, as well as business strategy (or greed, depending on which side you are on).
Level 3 has elected to frame this as a net neutrality issue and take the dispute to the press and the court of public opinion, but it's probably about a different question: who will pay, and how much will each party pay, for the infrastructure required to deliver content from L3's customers to Comcast's customers?
If viewed in this light, this peering agreement renegotiation is Comcast pushing back some of the costs on L3 and/or it's customers (i.e. streaming video providers). The other alternative would be to raise it's own customer rates.
It's possible that Comcast is being greedy and leveraging their position to rent-seek, but it's also possible that the traffic mix that resulted in the previous peering agreement is now significantly out of whack, justifying this renegotiation. It's definitely not black and white, and this is only one side of the story.
[1] http://en.wikipedia.org/wiki/Peering
[+] [-] ohashi|15 years ago|reply
[+] [-] joezydeco|15 years ago|reply
"Netflix (NASDAQ:NFLX) announced Thursday (Nov 11th) that it has signed a multi-year deal with Level 3 (NASDAQ: LVLT) for Level 3 to become a primary content delivery network (CDN) provider for the online movie rental company."
[+] [-] jrockway|15 years ago|reply
Sadly, I'm beginning to envision the future of the Internet as a bunch of point-to-point VPN connections to trusted sites, almost exactly like UUCP or the BBSes of days past. VPNs are already essential if you live in certain countries (China, the US if COICA had passed, etc.) or want to BitTorrent your TV shows without getting sued for 30 million dollars. Someday, it might be required just to watch Netflix or search with Google. Sad.
In what world do 65536 octets representing a video cost more to deliver than 65536 representing an email? A world where greedy ISPs can tell the difference.
[+] [-] hartror|15 years ago|reply
[+] [-] wmf|15 years ago|reply
[+] [-] jodrellblank|15 years ago|reply
In what world can you extract more cash from 65536 octets representing a video than 65536 octets representing an email? A world where end users value them differently.
[+] [-] absconditus|15 years ago|reply
I do not understand how Comcast had leverage in that situation. Had Level 3 told them no then Comcast customers would have been affected. Why is Comcast even negotiating with Level 3 and not the content providers?
[+] [-] jonknee|15 years ago|reply
I think they went for Level3 because average users have no idea who they are. Attacking Netflix gets you in the NY Times. Attacking Level3 gets you on a BusinessWire press release.
[+] [-] wmf|15 years ago|reply
[+] [-] ghshephard|15 years ago|reply
Peering Dispute.
"Comcast on Monday rebuffed the notion that the new fees were related to Netflix by saying that the type of traffic distributed by Level 3 was irrelevant. Joe Waz, a senior vice president at Comcast, says it has had a peering agreement with Level 3 to swap traffic fairly evenly. Now Level 3 is sharply increasing its traffic, he said, while resisting a commercial agreement to pay for that.
Comcast is “already carrying huge amounts of video to our high-speed Internet customers every day through commercial arrangements, and it seems to be working for everybody else,” Mr. Waz said. “Level 3 is trying to change the rules of the game.”"
[+] [-] woodrow|15 years ago|reply
http://blog.comcast.com/2010/11/comcast-comments-on-level-3....
[+] [-] drusenko|15 years ago|reply
This kind of thing happens all the time, relatively speaking. There's really nothing to care about here.
If you get upset over this, you're just playing into Level 3's hand. They're exploiting the "net neutrality" concept to try to get a better deal out of Comcast, even when this has nothing to do with it.
How extremely disappointing of them.
[+] [-] jaysonelliot|15 years ago|reply
Something concrete and egregious needed to happen before the average consumer would understand what it really means to allow companies to filter the type of traffic that people are allowed to get through their pipes.
[+] [-] maqr|15 years ago|reply
Comcast is going to isolate the content upstream so that consumers won't ever know that there was other content accessible, just like they don't know about the TV channels that Comcast doesn't carry.
[+] [-] smutticus|15 years ago|reply
[+] [-] NathanKP|15 years ago|reply
[+] [-] Terretta|15 years ago|reply
If Comcast blocked Level(3)'s Netflix hosting, customers would still get streams, just pulled from Akamai and Limelight.
I'd guess that Level3 is counting on "Tier 1 backbone" peering agreements to make delivering this bandwidth affordable while avoiding having to negotiate with each cable co separately, while Akamai and Limelight have direct agreements with the cable companies already.
If this is what's going on, that would give Comcast leverage.
[+] [-] akozak|15 years ago|reply
[+] [-] karzeem|15 years ago|reply
In the long term, it's obviously disastrous. At some point, companies will have to refuse en masse to pay the protection money. In the areas where it has a monopoly, Comcast could stomach losing Netflix, but not another 10 or 20 big sites on top of it. There are antitrust issues with collective refusal to pay, but there's got to be a way around that.
[+] [-] bbatsell|15 years ago|reply
While certainly not an ideal situation, I could easily see Netflix going after Comcast with highly-visible notices to their Comcast customers rather than pay them off and set a very, very bad precedent.
[+] [-] iopuy|15 years ago|reply
[+] [-] kevinpet|15 years ago|reply
[+] [-] danielnicollet|15 years ago|reply
[+] [-] kooshball|15 years ago|reply
[+] [-] tsotha|15 years ago|reply
[+] [-] 10smom|15 years ago|reply
[+] [-] cedsav|15 years ago|reply
[+] [-] donaldc|15 years ago|reply
How many broadband ISPs are there in your area? In many areas, there are no more than two. If they both adopt this policy, without net neutrality you're out of luck.
[+] [-] marklabedz|15 years ago|reply
[+] [-] pyre|15 years ago|reply
[+] [-] fennecfoxen|15 years ago|reply