As a lifelong sysadmin and former ISP person, I would have never thought that I would be impressed with Comcast. However, after I lost a sweet free colo deal (company I used to work for, got sold) a few years ago, I decided to move my main system back to the house. I signed up for Comcast Business, because AT&T's only "business" option with static IPs involved 1:1 NAT through their crappy 2Wire-brand gateway.
Anyway, years later - I'd give Comcast Biz a score of 95/100. When I called to get PTR records added for my five static IPs, I ended up on the phone with the guy who was typing them in. For the very few times I've had to call them (less than once a year), I've never had a tech try to run me through a "script", or argue that I didn't know what I was talking about when I tell them I've already done all the debugging and the problem was on their end.
Had upstream issues once (even to their own speedtest server, so it wasn't going outside of the Comcast network), and after a couple of calls I woke up one Saturday morning to two techs and a manager on my front porch. "We brought an unlocked modem and everything we might need, not leaving until this is fixed." They were at my place for about an hour, then went down the road to fix something on a junction box, and I had full speeds back.
Couldn't get them to leave that uncapped modem, though...
The service you get from a business tier is nothing like the service you get from residential. In Australia I managed a thousand or so ADSL connections with TID [1] and the support was excellent. Telstra is much maligned in Australia, but it's not like they can't provide good service.
Comcast Business Tier support had been decent. However, now they threaten a $99 service call charge if they can't replicate the issue or it goes away by the time the tech shows up. Response time is usually within a day but for something up the network a bit they'll just throw blame at the customer and bill them.
Otherwise, I will say that lately their commitment to keeping equipment up on the east coast has been admirable. My house has been without power since Friday but they threw generators on all the fiber nodes' power supplies to keep things up. pic: https://pbs.twimg.com/media/DXjUdljXUAElsRX.jpg
My current place of employment has Comcast. We have a major outage every couple months during which all work ceases. I rely on an Internet connection for everything I do professionally so I stay away from cable because stability > speed.
We have a few offices in the USA and the offices are connected to Comcast. I was really surprised how good they were after reading all the blog posts about how bad they were. In both locations they offered us the best service (Sip trunking and internet) at the best price. Not only that, they also had to shell out for a major build to one of our facilities without us flipping any part of the bill. When we've had problems, required changes, they were really fast and competent. I can't speak for their consumer division but I would recommend them based on my experiences.
I'd be very interested in reading about what you're doing with your home network, and how much you're paying for their services.
When I last looked, Comcast charged around $200/mo for 250Mbps down / 20Mbps up, unlimited bandwidth, and a static IP. At the time it seemed too much, but now I pay around $150 for for 130Mbps down / 13Mbps up, unlimited bandwidth, and a dynamic IP. So I might have to give it a bit more serious consideration.
I also have Comcast Business, but I'd give them a 40/100. (But most of that score is because the service usually works.)
> I've never had a tech try to run me through a "script", or argue that I didn't know what I was talking about when I tell them I've already done all the debugging and the problem was on their end.
I've had two major outages:
* One outage where IPv6 connectivity was completely lost, and IPv4 connectivity was completely unaffected. I noticed this immediately, because I don't use Comcast's DNS servers; instead I use Google's, over IPv6 exclusively, so it was immediately apparent.
* One outage where IPv4 SYN packets would go unanswered, however, once answered, the connection was quite stable. IPv6 was completely unaffected.
In both instances, the tech on the phone insisted that "my signal strength" was "very weak", and that they'd send a tech out to fix that right away. The representatives both time were unable to comprehend — let alone answer — my question of "how would signal strength only affect one of IPv4/v6? Do you think the copper can somehow discriminate which protocol is in use?" In the end, the representative adjusted by strength until it was supposedly awesome, and in neither case did the problem get resolved at that point, of course.
It took much insisting to get them to look at their own network in the first case, and to replace my modem in the latter case. (I have no idea what went wrong in the latter case, but my guess is "something with NAT" and/or the IPv4 hardware. Regardless, replacing the modem did fix the issue.)
I've also tried — and have yet to convince them of this: every single one of their black Comcast-branded modems will forget the 2 GHz WiFi WPA-PSK key on any sort of power cycle or reset, and revert back to the default, but only for the key. (All other settings, such as port forwardings or the SSID for the AP remain intact.) This, I believe, only happens if you use an actual key; if you use a password from which a key is derived, you won't be affected.
They've also tried to upsell me during outages … which, really? Why do companies do that?
I did at least finally get a representative that was human enough to laugh when I told them that the modem was doing the "blinken light dance". (The LEDs on the front panel go through a sequence as the device boots and acquires a connection.)
> When I called to get PTR records added for my five static IPs,
I would love to have this … I just didn't think I could request such a thing without the hell of trying to explain what a PTR record is.
I have a similar issue - very slow download - with my ADSL access provided by SFR in France.
I did similar tests, by downloading the same file from servers hosted at different providers, and the bandwidth was always very slow, except when I changed the HTTP port from 80 (the standard) to an arbitrary chosen number! This change alone made me able to use the full bandwidth provided by my physical link! It was the proof that my ISP was running some kind of traffic shaping, which is a shame.
No such issue over here (SFR FTTH), bandwidth is extremely stable and through the roof.
Free SAS is absolutely terrible in that regard. I ran a few experiments (at home - when I was still a customer - and at friends') over the years similar to that blog post, and starting 17:00 things get throttled down to death due to clogged pipes. It's almost on the clock. The behaviour is immediately obvious to anyone having some background in network engineering, and support is tone deaf to the issue.
> sends the user to run the same script on the control server, which we chose to host with Linode
Just went through a 3 month long issue with Linode, slow download speeds at peak hours on Cox cable. They said to call Cox, Cox escalated me to a guy who did not understand mtr tool. On Linode's end, they would not acknowledge any issue & burned up many hours of time trying to convince me there was no issue.
Ended up switching to Digital Ocean since Linode refused to call Cox on my behalf after I failed to get through. We also had the slowness issue with Linode with users in Dallas, where our servers were located. I suspect this will only get worse as net neutrality is repealed. I'm not saying these ISPs are slowing things on purpose, but if they do so accidentally, then imagine if they have green light to do so on purpose.
Although you very well might be, do not be tempted to stop reading after you get to the sentence, "The Panic web servers have a single connection to the internet via Cogent."
Believe it or not, it's not (just) Cogent this time. It's Comcast, too. Or, more specifically, saturated peering links between them.
To the others: this will likely not come as a surprise to any neteng who has had to deal with either of them.
That was interesting. I do believe that's the first time I've heard a positive story about Comcast. It's amazing how much cooperation has to happen to make the Internet work.
Is this really a positive story about Comcast? I suppose you can say all's well that ends well, but that they were collateral damage in Comcast's attempts to make Netflix double pay for traffic in the first place more than cancels that out, in my opinion.
I know it's completely anecdotal, but I've been with Comcast for two years now and I've always had a good customer service experience with them. That said, I'm in an area with a fair amount of ISP competition, so it's possible they're more careful here.
> The Panic web servers have a single connection to the internet via Cogent.
Say no more. Hope this company gives their NetOps department the requested budget next time. Would have been extra credit if they started monitoring their download servers for TCP retransmits and grouping it by netblock to look for trends (or just move it to the cloud where it's someone else's problem)
I've been fighting dreadful peering between CenturyLink and a variety of endpoints. I can saturate my gigabit connection on speedtest.net with a directly-peered test host, but YouTube videos at peak times of day buffer at 720p, and downloading from any USA OpenBSD mirror (of which there are only a few, and none in Seattle) nets me about 4mbps, compared to 450mbps or so I can get on a Comcast connection.
What's funny is that using a VPN or tunnel to which CL has a good connection resolves the issue, because the VPN/tunnel host has better peering than CL does.
The difference between Panic's anecdote and mine, is that CenturyLink support won't even acknowledge the issue exists.
I live in Seattle and use Centurylink and experience none of this. My children watch youtube all evening with no degradation of quality, even while my wife and I watch netflix. I have never experienced slowness with centurylink -- I wonder if there is some bad hop that the vpn is avoiding?
Many US ISPs have inadequate peering with YouTube and Netflix. I had the same issue with TWC and I've seen reports of it with other providers.
One thing to try is using different DNS servers. That may give you a different server with less traffic. If that doesn't work a VPN, as you mentioned, will.
The people downvoting you don't have experience with transit or Cogent - they are literally the worst peer. Their company motto is to undercut everyone else on price and oversaturate their peers as much as possible. They are a horrible provider. They have been de-peered by large providers like Layer3 many times due to flagrant abuse.
That said the price is great and I had a large presence at Cogent's Herndon location for the better part of a decade. Terrible service but price was so great, couldn't resist.
I'm not sure that's something that you can take away from the article, Cogent was performing well on other ISPs, Comcast was not providing enough bandwidth for Cogent due to a disput external to Panic.
While we can't know I'd be willing to be the "unspecified" traffic re-engineering was to prioritize Cogent's Panic traffic specifically. Comcast is still the bad guy.
> We colocate our own servers, rather than using AWS or any other PaaS, and we also don’t currently use a CDN or any other cloud distribution platform.
I wonder if Panic would never have encountered this slow download problem if they did use AWS or another PaaS.
> And then, the craziest thing happened… They wrote back quickly.
Now they are responsive, because of the whole Net Neutrality repeal backlash. They are scared to provide factual evidence of foul play - it will help the courts to blast FCC for their decision.
But if they win in courts, Comcast and other crooks will definitely double down on starving peering, as expected to extort money. And obviously, they won't ever write back on this topic or will write that there is nothing wrong on their side.
The only way to prevent that peering extortion by Comcast and the like, is to have explicit rules that forbid it.
They are, but even the larger players get into peering disputes with eyeball networks. Level3 and Comcast had a huge peering spat a few years ago (via Netflix). Everyone's ports run hot with China Telecom and China Unicom.
It's almost always political (revenue) problems and rarely technical.
That said, I wouldn't pick up Cogent with their negative attitudes towards IPv6.
Speaking of slow downloads, when I save a webpage in Chromium, it seems as if it downloads the page byte by byte, reporting progress after every byte. And another quirky thing is that if I close the tab during a download, then the download is cancelled. Did anyone else notice this?
For some reason the byte counter represents number of saved files when you do a "Save complete web page" in Chromium. It still displays like "4/53 B, 5 secs left". Weirdly sloppy.
I'd love a service that gave me the right contact for the right problem in all the businesses possible. Finding the right person while googling seems like a matter of luck.
[+] [-] mrbill|8 years ago|reply
Anyway, years later - I'd give Comcast Biz a score of 95/100. When I called to get PTR records added for my five static IPs, I ended up on the phone with the guy who was typing them in. For the very few times I've had to call them (less than once a year), I've never had a tech try to run me through a "script", or argue that I didn't know what I was talking about when I tell them I've already done all the debugging and the problem was on their end.
Had upstream issues once (even to their own speedtest server, so it wasn't going outside of the Comcast network), and after a couple of calls I woke up one Saturday morning to two techs and a manager on my front porch. "We brought an unlocked modem and everything we might need, not leaving until this is fixed." They were at my place for about an hour, then went down the road to fix something on a junction box, and I had full speeds back.
Couldn't get them to leave that uncapped modem, though...
[+] [-] taspeotis|8 years ago|reply
[1] https://www.telstra.com.au/business-enterprise/solutions/net...
[+] [-] joecool1029|8 years ago|reply
Otherwise, I will say that lately their commitment to keeping equipment up on the east coast has been admirable. My house has been without power since Friday but they threw generators on all the fiber nodes' power supplies to keep things up. pic: https://pbs.twimg.com/media/DXjUdljXUAElsRX.jpg
[+] [-] protonfish|8 years ago|reply
[+] [-] dangerboysteve|8 years ago|reply
[+] [-] TheAceOfHearts|8 years ago|reply
When I last looked, Comcast charged around $200/mo for 250Mbps down / 20Mbps up, unlimited bandwidth, and a static IP. At the time it seemed too much, but now I pay around $150 for for 130Mbps down / 13Mbps up, unlimited bandwidth, and a dynamic IP. So I might have to give it a bit more serious consideration.
[+] [-] deathanatos|8 years ago|reply
> I've never had a tech try to run me through a "script", or argue that I didn't know what I was talking about when I tell them I've already done all the debugging and the problem was on their end.
I've had two major outages:
* One outage where IPv6 connectivity was completely lost, and IPv4 connectivity was completely unaffected. I noticed this immediately, because I don't use Comcast's DNS servers; instead I use Google's, over IPv6 exclusively, so it was immediately apparent.
* One outage where IPv4 SYN packets would go unanswered, however, once answered, the connection was quite stable. IPv6 was completely unaffected.
In both instances, the tech on the phone insisted that "my signal strength" was "very weak", and that they'd send a tech out to fix that right away. The representatives both time were unable to comprehend — let alone answer — my question of "how would signal strength only affect one of IPv4/v6? Do you think the copper can somehow discriminate which protocol is in use?" In the end, the representative adjusted by strength until it was supposedly awesome, and in neither case did the problem get resolved at that point, of course.
It took much insisting to get them to look at their own network in the first case, and to replace my modem in the latter case. (I have no idea what went wrong in the latter case, but my guess is "something with NAT" and/or the IPv4 hardware. Regardless, replacing the modem did fix the issue.)
I've also tried — and have yet to convince them of this: every single one of their black Comcast-branded modems will forget the 2 GHz WiFi WPA-PSK key on any sort of power cycle or reset, and revert back to the default, but only for the key. (All other settings, such as port forwardings or the SSID for the AP remain intact.) This, I believe, only happens if you use an actual key; if you use a password from which a key is derived, you won't be affected.
They've also tried to upsell me during outages … which, really? Why do companies do that?
I did at least finally get a representative that was human enough to laugh when I told them that the modem was doing the "blinken light dance". (The LEDs on the front panel go through a sequence as the device boots and acquires a connection.)
> When I called to get PTR records added for my five static IPs,
I would love to have this … I just didn't think I could request such a thing without the hell of trying to explain what a PTR record is.
[+] [-] joncrane|8 years ago|reply
[+] [-] ngrilly|8 years ago|reply
I did similar tests, by downloading the same file from servers hosted at different providers, and the bandwidth was always very slow, except when I changed the HTTP port from 80 (the standard) to an arbitrary chosen number! This change alone made me able to use the full bandwidth provided by my physical link! It was the proof that my ISP was running some kind of traffic shaping, which is a shame.
More information if you read French: https://twitter.com/ngrilly/status/756453318113783809?s=19
[+] [-] lloeki|8 years ago|reply
Free SAS is absolutely terrible in that regard. I ran a few experiments (at home - when I was still a customer - and at friends') over the years similar to that blog post, and starting 17:00 things get throttled down to death due to clogged pipes. It's almost on the clock. The behaviour is immediately obvious to anyone having some background in network engineering, and support is tone deaf to the issue.
[+] [-] joshribakoff|8 years ago|reply
Just went through a 3 month long issue with Linode, slow download speeds at peak hours on Cox cable. They said to call Cox, Cox escalated me to a guy who did not understand mtr tool. On Linode's end, they would not acknowledge any issue & burned up many hours of time trying to convince me there was no issue.
Ended up switching to Digital Ocean since Linode refused to call Cox on my behalf after I failed to get through. We also had the slowness issue with Linode with users in Dallas, where our servers were located. I suspect this will only get worse as net neutrality is repealed. I'm not saying these ISPs are slowing things on purpose, but if they do so accidentally, then imagine if they have green light to do so on purpose.
Linode also uses Cogent, FYI.
[+] [-] jlgaddis|8 years ago|reply
Although you very well might be, do not be tempted to stop reading after you get to the sentence, "The Panic web servers have a single connection to the internet via Cogent."
Believe it or not, it's not (just) Cogent this time. It's Comcast, too. Or, more specifically, saturated peering links between them.
To the others: this will likely not come as a surprise to any neteng who has had to deal with either of them.
[+] [-] mschuster91|8 years ago|reply
[+] [-] Maultasche|8 years ago|reply
[+] [-] e40|8 years ago|reply
[+] [-] gnode|8 years ago|reply
[+] [-] supergarfield|8 years ago|reply
[+] [-] jerkstate|8 years ago|reply
Say no more. Hope this company gives their NetOps department the requested budget next time. Would have been extra credit if they started monitoring their download servers for TCP retransmits and grouping it by netblock to look for trends (or just move it to the cloud where it's someone else's problem)
[+] [-] culturestate|8 years ago|reply
[+] [-] tbyehl|8 years ago|reply
http://customer.xfinity.com/help-and-support/vp-contact-form
In my experience, the people responding to those tickets get shit done.
[+] [-] Serow225|8 years ago|reply
[+] [-] darkengine|8 years ago|reply
What's funny is that using a VPN or tunnel to which CL has a good connection resolves the issue, because the VPN/tunnel host has better peering than CL does.
The difference between Panic's anecdote and mine, is that CenturyLink support won't even acknowledge the issue exists.
[+] [-] tkone|8 years ago|reply
[+] [-] driverdan|8 years ago|reply
One thing to try is using different DNS servers. That may give you a different server with less traffic. If that doesn't work a VPN, as you mentioned, will.
[+] [-] ohf|8 years ago|reply
[+] [-] Pilfer|8 years ago|reply
[+] [-] jasongill|8 years ago|reply
That said the price is great and I had a large presence at Cogent's Herndon location for the better part of a decade. Terrible service but price was so great, couldn't resist.
[+] [-] chuckgreenman|8 years ago|reply
While we can't know I'd be willing to be the "unspecified" traffic re-engineering was to prioritize Cogent's Panic traffic specifically. Comcast is still the bad guy.
[+] [-] stevoski|8 years ago|reply
> We colocate our own servers, rather than using AWS or any other PaaS, and we also don’t currently use a CDN or any other cloud distribution platform.
I wonder if Panic would never have encountered this slow download problem if they did use AWS or another PaaS.
[+] [-] toomuchtodo|8 years ago|reply
[+] [-] joshribakoff|8 years ago|reply
[+] [-] NKCSS|8 years ago|reply
[+] [-] bewo001|8 years ago|reply
[+] [-] tgtweak|8 years ago|reply
It's almost mandatory unless you have 3-4 POPs strategically placed geographically.
[+] [-] shmerl|8 years ago|reply
Now they are responsive, because of the whole Net Neutrality repeal backlash. They are scared to provide factual evidence of foul play - it will help the courts to blast FCC for their decision.
But if they win in courts, Comcast and other crooks will definitely double down on starving peering, as expected to extort money. And obviously, they won't ever write back on this topic or will write that there is nothing wrong on their side.
The only way to prevent that peering extortion by Comcast and the like, is to have explicit rules that forbid it.
[+] [-] voltagex_|8 years ago|reply
[+] [-] scurvy|8 years ago|reply
It's almost always political (revenue) problems and rarely technical.
That said, I wouldn't pick up Cogent with their negative attitudes towards IPv6.
[+] [-] Hello71|8 years ago|reply
Because Panic doesn't compete with Xfinity.
[+] [-] amelius|8 years ago|reply
[+] [-] vgf|8 years ago|reply
[+] [-] sirtel|8 years ago|reply
[+] [-] mynewtb|8 years ago|reply
[+] [-] hiccuphippo|8 years ago|reply
[+] [-] rsre|8 years ago|reply