top | item 12763501

Possible Vendetta Behind the East Coast Web Slowdown

187 points| whiskypeters | 9 years ago |bloomberg.com

198 comments

order
[+] jerf|9 years ago|reply
For a long time, I've wondered what would finally be the Securitypocalypse, the thing that finally caused our industry as a whole to take security seriously. These IoT DDoS attacks are as good a candidate as any I've seen in a long time. They are fundamentally very difficult to fix in light of the non-updateability of many of these devices, and this is only the beginning, because the IoT has hardly begun to develop. And in the short-term, I'm not sure I see any hope, because the forces that make people throw out cheap devices with broken firmwares with no update capability aren't going away.

If we could somehow mandate that these devices were supported with firmware updates for the indefinite future, that would simply destroy the entire market. And you can't do that, because even the devices created by an entity that no longer exists and didn't sell its IP to anybody else will eventually be enough to do these DDoSes, if they aren't already.

[+] gtrubetskoy|9 years ago|reply
These attacks are mostly possible because of the complacency of operators at many sites and companies. This is not a new problem and many of RFC's talk about methods for preventing and mitigating them, but most people don't care and prefer to just outsource everything to a single provider, which becomes the weakest link.

The Internet wasn't envisioned with a single email provider, single DNS provider, single app container provider. (Ok, for most of these you have two, sometimes three choices, but still, that is too few). The centralization makes everything very vulnerable - imagine what would happen when Gmail is knocked out for a day.

[+] NKCSS|9 years ago|reply
It's easy to fix; back in the day when a machine was infected; an ISP would just block outgoing traffic, contact line owner and re-enable when the issue is resolved.
[+] seanp2k2|9 years ago|reply
Yep, and manufacturers have not much incentive to update firmware for a device which is not their latest greatest or update firmware while not adding more features to help them sell more. Security isn't a feature that the vast majority of consumers would pay extra for or know how to verify anyway. There was plenty of demand for that one "unhackable" android phone, but I'd be blown away if it wasn't 100% snake oil.

My prediction is that it'll get worse before it gets better and that these type of botnets will be around for at least 5 years. Look at what happened to unsecured-by-default routers, android phones, Windows PCs, cars...the way consumers will get more secure stuff is by manufacturers being publicly embarrassed / sued over problems until caring about security makes business sense, then they'll have it in their hands when their old insecure gadgets die.

My cynical side side thinks this will be a problem until all the old endpoints supporting these insecure things are shut down eventually in 5-10 years.

[+] JoelBennett|9 years ago|reply
There'd be something ironic about a manufacturer's website being made unavailable because of a DDoS caused by their own poorly secured devices.
[+] trhway|9 years ago|reply
>They are fundamentally very difficult to fix in light of the non-updateability of many of these devices

as you proved, fixing the situation by fixing the devices wouldn't be a feasible approach. The traffic from those devices is carried by ISPs and this is there this traffic should be stopped. To me the situation reminds about email spam. We didn't get rid of spammers, instead the email traffic is analyzed and dealt with accordingly. I'm sure that ISPs easily see the patterns of such massive DDoS attacks and could just drop (or throttle down into oblivion, like 100s times down) the participating traffic.

[+] HipHopHacker|9 years ago|reply
> For a long time, I've wondered what would finally be the Securitypocalypse, the thing that finally caused our industry as a whole to take security seriously.

Nothing. If the economic system revolves around capital's valorization of itself, security is a distraction from that. I have to spend five seconds typing my password in every time I sit at my desk? I can't just easily e-mail this executable file to my co-worker and have them run it? My desktop is locked down by the desktop admins to prevent me being able to do this, and many other things? Every implementation of security costs money for the personnel to do it and possibly the product cost. Plus any lost productivity it might cause (15 seconds to type in a password each time one sits at their desk, compounded).

Donn Parker wrote one of the first books on computer security in 1976, Crime by Computer. The opening words are as apt for corporate security now as it was then. The #1 fear for the corporate manager are the employees of that company. They are the ones with the greatest control over the means of production, so to speak, even more than the managers themselves who are de jure in charge, but are de facto one step away from actual control. Look at how much access someone like Snowden had at Booz Allen.

Obviously, if all products have wide open holes, script kiddies will be able to get control. Some minimal security will always be done to stop this sort of thing. On the other hand, one (or better yet, several) dedicated people who want to get past some security arrangement can almost always get in. Even if the firewall is supposedly impenetrable, the wifi or the building security or the social engineering credulity of employees or something will be there. There will be some weak link in the chain. Especially for a company that needs to make a profit.

The real security is that semi-intelligent, persistent agents that seek to access and control systems without authorization are lacking. Things depend on the conditions that cause this to rise or diminish. Because once it rises, there is little that can be done. I forget who said that the czar's Russian Okhrana was one of the largest, most extensive security forces that existed. That meant little when Russia began collapsing in 1916 though - all it meant was that they were even more aware that virtually everyone in the country was becoming the czar's enemy.

Securitypocalypse events due result in business and government putting more focus on security for a while, but time moves on, and attention drifts back to the main focus. These things go in waves, and total security is never something of the highest priority.

[+] egypturnash|9 years ago|reply
I am a non-programmer who reads HN and keeps up with tech news in general.

And every time I read about the IoT botnet, my immediate response is to look around my apartment at my Internet-connected lights, and wonder if they're part of it.

How can I find this out?

Is anyone making a tool that a non-technical user can run to squint at their network and look for evidence of Mirai, or anything else trying to take advantage of this niche?

There are plenty of tools with a reasonably simple interface that will tell me if my laptop/desktop computer is infected with something. But what can I use to diagnose the health of all of the other computers proliferating around my house?

How can a non-technical user easily monitor the overall health of their connected household? Is this a project anyone is building? Because I think it's definitely something that needs to exist now.

[+] egypturnash|9 years ago|reply
Looking over all the replies this comment received, I think my plan for seeing if my apartment's Internet Things are on any botnet is going to be "bribe that security researcher I flirt with sometimes to visit my place and run some tests". Which is not really a solution that scales, either for that friend, or for people who don't happen to run in the kinds of circles where that's someone they could conceivably trade favors with.

And it's probably not gonna get any better any time soon, either. Because I'm not sure there's a money stream in making this something a non-programmer can do. And maybe there shouldn't even be a money stream in this - maybe there should just be huge-ass fines to motivate as many people as possible along the chain from "my Internet Thing" to "the Internet" to include a white/grey hat or three on their team very early in the design process of making their camera/light bulb/pacemaker/router/modem/whatever. Although if someone reading this can figure out a way to get a money stream out of making it a lot easier to see the health of your home's devices, and keep them safe, that might be a decent YC app for you.

How do we add an immune system to the Internet Of Things? Because we sure as hell need one.

[+] rabboRubble|9 years ago|reply
My reading of Krebs On Security (krebsonsecurity.com), Mirai scans for factory default passwords or hardcoded default admin credentials. Going at this as a non-technical person, I would:

* Inventory all IOT devices in your possession.

* Find the device manuals and make sure you've changed the default password(s). Note there may be devices where it appears you've updated, but that have secret credentials you can't modify.

* Make note of which of your devices do not have an obvious way to change the factory default password.

* Keep an eye out for lists of devices that are known problems, here is one such sample list: https://blog.sucuri.net/2016/09/iot-home-router-botnet-lever...

* Check each manufacturer to see if they have issued a firmware upgrade to address security issues. Apply update.

* Think about retiring devices that appear on the "bad" hardware lists or the devices with unchangeable factory defaults.

Hope this helps.

[+] smartbit|9 years ago|reply
Dowse is trying to help you out http://dowse.eu/#sec-2-2 Dowse is a transparent proxy facilitating the awareness of ingoing and outgoing connections, from, to, and within a local area network. ... Dowse communicates with users in various ways: via a web interface, but also pushing messages via audio (synthesized speech), Bonjour and simple apps interfacing with personal mobile devices.

You can even hook up Dowse to your TV set and show a live animation of where on the your internet your devices connect to https://youtu.be/vquh3IXcduc?t=74

[+] dmourati|9 years ago|reply
The best place to do this is at your border. You probably have a cable modem or router or some such that connects your home to the internet. You would typically install software known as IDS (Intrusion Detection System) such as Snort there and look for anomalous traffic.

As for a non-technical solution, it will be difficult to implement. It requires some computer know how and time. Such a secure device could be created or better yet offered by the manufacturers of the modems/routers frequently deployed in homes.

[+] pmlnr|9 years ago|reply
For non-technical users, I'd suggest the following:

Turn off the devices you don't want to check; leave only those up you want to investigate.

Visit your router on the web interface and see if there are any graphs are possible to check for things like requests/second or packets/second. If it's really high while you're not actively doing anything, that's a clue.

Visit the UPnP settings on the router. If there are ports listening for incoming internet connections, turn the UPnP off on the router.

If you can SSH to your router, log in and start collecting data with tcpdump[1]. This later can be analyzed with tools like wireshark[2], which can show you per protocol what is going in. Most of the botnets use telnet- or IRC-like interfaces which are, in most cases, plain text commands, so it's possible to spot them.

Apart from this: reset everything to factory and change all the passwords before letting anything on your network.

[1]: https://www.linux.com/blog/tcpdump-tutorial-beginners

[2]: https://www.wireshark.org/docs/wsug_html_chunked/ChapterIntr...

[+] rfrank|9 years ago|reply
WireShark is the first thing that jumps to mind, although I'm not sure if easy for non-technical users is the way I'd describe it.

https://www.wireshark.org/

[+] avip|9 years ago|reply
To be part of Mirai network, your device needs to have telnet access open to the world AND use default factory credentials (which in turn must be on the lines "admin admin" or "root root").
[+] Retr0spectrum|9 years ago|reply
I might be wrong, but in the case of Mirai I'm fairly sure you're safe if all your devices are behind NAT.
[+] gorbachev|9 years ago|reply
Here's a better article from Mr. Krebs:

https://krebsonsecurity.com/2016/10/ddos-on-dyn-impacts-twit...

Personally I think his case is pretty convincing.

[+] brightball|9 years ago|reply
From the article:

"Last month, a hacker by the name of Anna_Senpai released the source code for Mirai, a crime machine that enslaves IoT devices for use in large DDoS attacks. The 620 Gbps attack that hit my site last month was launched by a botnet built on Mirai, for example."

I repeatedly hear people refer to IoT devices that are notoriously difficult to update...yet this Mirai code is technically able to access millions of devices and bend them to its will.

So what I'm wondering is just, what prevents the good guys from using Mirai to slurp down every available device to patch the vulnerability that allowed Mirai to work in the first place?

It seems like if vulnerabilities in these devices can destabilize the entire internet that it should be perfectly viable as a response to actively look for those vulnerabilities, patch/minimize them and notify their creators of the issue.

[+] dom0|9 years ago|reply
Guess what, Krebs' site also receives a spanking at the moment. (Given that it's hosted by Google I find it highly unlikely to go down under normal traffic)
[+] dates|9 years ago|reply
krebs is loading reaaaaal slow for me...i wonder if its related? or just a lot of people linking to it today.
[+] bcheung|9 years ago|reply
I know the TTL is set really low for a lot of DNS entries but this recent outage got me wondering if it makes sense for servers further down the chain to hold onto it for longer than the TTL, honor it when they are able to get a new DNS entry within a reasonable amount of time, but fall back to the "expired" version if the authoritative server is not reachable.

I'm wondering what would be the negative consequences of this and if they outweigh the benefit of being more resilient to these types of attacks.

[+] elmigranto|9 years ago|reply
No luck with Google DNS for me, but Yandex seems to work:

    77.88.8.8
    77.88.8.1
https://dns.yandex.ru
[+] drinchev|9 years ago|reply
Probably, but I would definitely avoid giving all my DNS resolutions to a *.ru domain.

The reputation of the government - shutting down access to websites that hurt them is kind-a no-go for me.

[+] CalChris|9 years ago|reply
Yeah, that worked for me in SV. But I'd rather not rely on anything Putin related.
[+] jpeg_hero|9 years ago|reply
bloomberg was down for me.

I had disabled adblock at their insistence...

i re-enabled adblock and I could get the article. hmmmm. maybe something about the 50 unrelated js calls?? perhaps?

[+] kakarot|9 years ago|reply
Unfortunately, forced firmware updating is an area our governments should not be mandating. That puts unnecessary strain on small companies and creates a larger gap that companies must cross to become commercially viable
[+] Analemma_|9 years ago|reply
> Unfortunately, forced firmware updating is an area our governments should not be mandating.

It absolutely is an area that governments should be mandating, because the problem is an externality. These attacks are a cost imposed on neither the producer nor the consumer of the device itself, and (apart from some highly speculative libertarian conjectures) the only things that can fix externalities are taxes, regulations, and lawsuits.

Lawsuits are infeasible in this case since we probably can't prove whose devices were involved in any given attack, so that leaves taxes and regulations - and the latter would be better so we don't have to go through the business of collecting taxes from manufacturers and then distributing them to the victims of attacks.

> That puts unnecessary strain on small companies

Clearly it isn't unnecessary, because I can't get to any friggin websites today. If small companies don't have the resources to update the devices, they shouldn't be building them in the first place.

[+] DashRattlesnake|9 years ago|reply
I don't think that's necessarily a bad thing. If a company doesn't have the resources to create secure products, then maybe it shouldn't be in that business in the first place.
[+] snovv_crash|9 years ago|reply
Just like with bridges. Getting certification by a professional engineer is just too much barrier to entry for small construction companies.

Edit: forgot the /s

[+] throwaway729|9 years ago|reply
Firmware updating isn't exactly a "hard tech" problem, even if it is hard to do right. I suspect we'll see some generic firmware update frameworks/solutions emerge in the coming decade, and at that point adoption will pick up rapidly because being able to push updates is good for business.
[+] Ar-Curunir|9 years ago|reply
In an age where security vulnerabilities can cause your thermostat to overheat your house and your smart lock to lock you out, maybe it'll be a good thing that companies that don't have good security practices and update mechanisms will be locked out of the IoT market.
[+] the_watcher|9 years ago|reply
Yea, it's a hard problem. While there's clearly a lot of vulnerabilities out there that emerge because it's cheaper to ignore security until you're large enough for a breach to be a big issue, forcing mandatory updates is a great way to discourage anyone from attempting to try something new. There might be a tipping point at which the costs of a breach outweighs the benefit, and maybe we've hit it already, but government mandates should be something we discuss cautiously and should prefer to avoid.
[+] davidhowlett|9 years ago|reply
I think that the negative externalities of poorly secured IoT devices scale linearly with the number attached to the internet whereas the cost of writing more secure software and keeping it updated scales much much more slowly with the number of installs. I think this means that the best solution is to have tiered levels of certification and regulatory burden based on the number of times a piece of software is installed. Ideally tiering would be done on the total bandwidth of all devices with a piece of software installed but this would be much more difficult to measure and enforce then counting installs.

If no product with less then X thousand installs has to deal with the regulatory overhead of certification then experiments and early stage companies are less likely to be squashed. I would also exempt open source software from having forced audits or minimum standards for security. This would have the side effect of encouraging more companies to publish their firmware open source which would also not be a bad outcome.

To prevent companies manufacturing lots of almost identical product lines each individually under the limit for audits I think it would also be necessary to count all products that share more then half their code as one product.

[+] ams6110|9 years ago|reply
Liability should be on the people who connect these things to the public internet. The owners of the devices. Like with cars, you have certain responsibilities and liabilities when you operate a potential dangerous machine on the public roads.

In the case of ISPs providing cable modems and routers and DVRs and other boxes to their customers, they should be responsible for keeping those secure.

If people start getting fines or sued over what their internet-connected devices are doing, they might stop connecting them to the internet, or shop more carefully for devices or providers that are secure.

[+] thesteverichey|9 years ago|reply
Any evidence this is using the IoT botnet that was reported on earlier this year?
[+] micaksica|9 years ago|reply
Mirai? With the source of that being public, there are probably quite a few Mirai botnets now.
[+] rrggrr|9 years ago|reply
These attacks are possible because the US Congress hasn't extended tort liability to manufacturers of software and network hardware. The full weight of the US products liability bar will quickly and rapidly motivate manufacturers to ship secure devices. The lack of accountability is enabling vulnerability.
[+] davidf18|9 years ago|reply
The failing here as in many cases such as a number of security breaches was a lack of investment. As someone with an engineering degree that worked as a VLSI design engineer, good engineering requires * backup systems *. This costs money that people don't want to spend. In some cases such as a startup they might be cash short, but many firms have the money but don't want to spend it ensuring that they have well engineered software that includes backups, up-to-date software and security upgrades, hiring (expensive) highly competent software engineers and consulting firms.

The mistake in this case was relying on one vendor for DNS. Amazon Route 53 would be a good alternate vendor for DNS, for example.

[+] patrickg_zill|9 years ago|reply
I think even basic home routers these days, have enough cpu power to handle egress filtering.

If you have an iot device, by its nature it only needs to connect to a few services and hosts.

The manufacturer can provide this in their docs, and give an automatic config url that the router uses to load its egress rules.

The rules to load are displayed and the user checks they are legit by comparing to the printed version in the manual, then clicks ok. Or something like that.

Rate limits in terms of packets per second, total bandwidth both instantaneous and over time, are set also.

[+] raverbashing|9 years ago|reply
Not only East Coast, Twitter can't be resolved in Ireland/UK right now (I assume the mobile app uses some kind of 'dns pinning' as that is working)
[+] woliveirajr|9 years ago|reply
I love those comments about IoT and who should be responsible for error-proof products, or ISP monitoring traffic, or ...

Internet, in the beginning, was even more insecure. Including the computers and OSes. There were less abuse because few had resources and knowledge. Read some old software and you'll find all bad designs in it. Software didn't become worst, it's just targeted with more knowledge and intensity.

[+] ilaksh|9 years ago|reply
DNS is actually fairly centralized the way it is actually used.

We need protocols and systems that are designed to be distributed from the outset.

[+] pc2g4d|9 years ago|reply
I always thought DNS had enough redundancy built-in that this sort of thing wouldn't really have much effect. But here I am unable to access websites, simply because name resolution isn't working. If my local DNS server were caching things longer there would largely be no issue.
[+] reacharavindh|9 years ago|reply
Perhaps a naive question, but Why can't a DNS provider identify such participants in a DDOS and ban their IPs forever?
[+] anotherevan|9 years ago|reply
Did any one else find the style of writing in this article really annoying? Things like using prefacing statements with "so-called" or putting terms in quotes to make them seem suspect.

e.g.s:

a so-called distributed denial-of-service (DDoS) attack

York said Dyn was “actively” dealing with a “third wave” of the attack.

[+] meira|9 years ago|reply
Not working, is bloomberg down too?
[+] gaur|9 years ago|reply

[deleted]