I've done some work in steel factories, though only for offline/closed systems that have no interaction with the main control code.
Factories like this one probably operate at around 60%+ capacity, so they'll be operating sometimes all day, sometimes all night. If you ever get the chance to visit, do so, even if you don't really care about how steel is made. The sheer scale of everything is amazing.
Everything is very big, very hot and if you have to hit the big red button, it costs a lot of money. Unscheduled downtime is very expensive. Steel tends to be workable when it's hot/molten and therefore pliable. If you suddenly stop a machine then you're left with solid steel in places you don't want it which takes a lot of time and effort to remove.
One of the common reactions to this story is "Why didn't they hit the emergency stop?" - the answer is because it costs an absolute fortune to do so.
i am working in a steel plant for over 20 years now, and it is easy to bash the security of those people.
but just some facts from my world :-)
first those plants are build for lifespans of over 30 years. general problem is 15(normal review time) years ago no one was thinking about network security as we thinking about it know . most businesses didn't even have a large internal network wich did include the production and were connected to the internet.
second you can't just shutdown this things. if you have to shutdown a blast furnace we are talking about minimum stand time of 5-7 days. calculate about 400k to 1m € per day on standstill cost. and that is only for the blast furnace. if the blast furnace is not running in some steel plant NOTHING will run. (e.G. hot rolling plants)
third there is no good solution on the market. if some of your guys would look into the software wich is sometimes running those large machines you would get sick to your stomach. As a more security focused person in my plant just to convince management to change the std admin passwords was a handful (well that changed like a year or two years ago). The thing is market decides what security is gonna be implemented. since there has now been a breach and a very expensive one most companies i am talking to are more focused on security now. The thing is they won't just throw away their software stack they worked on for 30 years. and reviewing software is hard and time consuming. so it will be interesting how this is developing.
I never understand why people need to connect industrial plants to the Internet. Do they actually need to control them over the Internet instead of on-site?
And, if they need to use the Internet on-site, can't they make an air gap and segregate the computers that can access the Internet from computers that can access the plant machinery?
I remember a rule a controls engineer once told me: never connect the plant to the Internet. Nothing clever, no humorous quip, no deep insight. Just don't do it. If you do need to get data out to the living world, and you will, then you carefully set up individual firewall rules just for the data system - which very much can not drive the process system.
This is where I begin ranting on the topic.
Why? Because plants aren't secure. They're meant to run all the time by people who may or may not know how to properly use a mouse. There will be passwords taped to monitors, systems that automatically log in to prevent start up delays, and authentication of the order that it-better-just-work-by-default. Under no circumstances should that damn system ever, ever be exposed to the outside world. Not by Ethernet, wireless, or flash drive. It's a young innocent facing the cruel, brutal internet; it's going to get hurt.
New plants are fancy and have highly trained workers with brilliant industrial IP wireless systems with state of the art VM servers and oh god did that guy just plug his phone charger into the damn wrapper HMI and now Windows media player has popped up and no one can acknowledge alarms (this partly why PCs tend to be in large metal boxes, that and dirt/water). No imagine that sort of silliness, but driven by the less savory from outside the intranet.
Just don't risk it. You'll never have a problem, and no one is ever going to care enough to hurt the plant... Except but for that one time when suddenly all the convenience and hubris won't bring back the machine that just slagged itself from some malicious command from outside.
They do not see themselves as targets. Their systems are likely bespoke, or at the very least obscure. And, more importantly, they have bigger problems to worry about, like operating their businesses.
As it becomes more clear that yes, someone will go to that trouble and it will have catastrophic consequences, you would hope that these things would get better.
More likely, someone will pay a "security consulting" company $100 million to run a Nessus scan and tell them to turn on Automatic Updates on their Windows infrastructure.
A better, more serious answer: always have two networks. Preferably physically separated (though I hear virtual networks are pretty ok with the right router equipment). The machines holler on one, and the administrative support on the other. I've seen what happens when it's even just tried to put both on the same layer, and it's inevitably some form of minor disaster. Not just because of security, mind, but because you really don't want your file transfer to a network drive to even slightly lag a sensor yelling back to the PLC about an interlock's state.
If you need to get on the process network, use a VPN, and only open to a machine that can't actually run equipment. A programming terminal may be made available to save costs so an integrator doesn't need to fly in for every support call, but these access points tend to require a VPN through at least two firewalls. (And even then, often you would still insist on them coming in person, for all manner of other reasons.)
OK, granted they may want to monitor the plant remotely. Then they could have a plant-connected machine dump UDP monitoring packets to an Internet-connected machine, and have the plant-connected machine block all incoming packets from the Internet-connected machine.
It is very unlikely that the process control network is connected to the Internet. However it is almost certainly connected to the corporate Intranet. Think about all of the metric data available on the process control network - that is needed by engineers for analysis, ERP systems for financials, asset management systems for maintenance etc. With an air gap, you can't do any of that in real-time.
The German document isn't that useful. It's just a general overview of computer security with anecdotes, not a technical analysis of this attack.
Interestingly, there was a cooling water leak and an emergency shutdown at a steel plant in Pakistan in October. That plant is still off line. That's probably unrelated, though.
Steel plants run for years without a shut down, so this was a large scale incident as the had to shut it down because of major damage.
Not related to the plant in Germany in any way, just to get you an idea how some other steel plants operate: C# WinForm based GUI control room app and Java based server app on Windows server. The server controls the various SPS. Several steel plants around the world were build with that software setup and it was not designed to be connected to the internet.
The register has speculation that it was a Thyssen Krupp plant in Brazil I susepct that if it had been actualy ingermany there might have been better security.
To do external monitoring, couldn't you have the computer for the plant display the information on a screen in a particular font and then an internet-connected computer read the video and OCR it?
I can't help but feel there's a rush to judgement here. If you read the article it clearly states that the Federal Office for Information Security (BSI) said, quoting the article:
"describing the technical skills of the attacker as “very advanced.”"
And
"not only was there evidence of a strong knowledge of IT security but also extended know-how of the industrial control and production process."
And HN rushes to judgement to quickly blame workers who can't use a mouse and Microsoft.
Yes, the average worker in a manufacturing plant is not a CS grad. It is the job of engineers to develop systems that are usable by, well, the target user.
Most Heart Surgeons don't have a CS degree. And based on meeting a number of them during the course of my business I am comfortable saying that quite a few of them are "computer challenged". Yet, most of us would not have a problem being on that operating table, yes, with a room full of computers, a good number of them running MS software and with an OR team that is likely to use the same "123456" password on everything.
In a hospital you have IT and engineers who setup an infrastructure medical professionals can use. The same is true of steel plants. Yes, there's probably a lot more older code in your average steel plant. I just don't think characterizing them as IT or security morons migt be fair.
The BSI characterized the attackers as sophisticated across disciplines. Let's not engage in senseless conjecture.
I've owned and operated a small manufacturing plant consisting mostly of what I call "big iron" CNC equipment. Things are seldom as simple as discussions on various fora on the 'net would like them to be. Yes, in my case I air-gapped the plant and even individual machines and remote monitoring was done through a separate network that had no command-and-control capabilities at all, just sensing and reporting. There was no way to jump from the sensing network to command-and-control of any one machine, much less the plant. Even if you were physically at the factory this was pretty much impossible. Nobody wants a CNC milling machine with a 30HP spindle controllable from the internet. People are not that stupid...even if they can't use a mouse.
I doubt it's another steel manufacturer but who knows? Maybe someone in the business with connections to black hats had some money to spare and said: Look what you can get going about this cyberwar stuff everyone is talking about...
Why does it have to be someone with a solid motive?
The companies attacked always go on about how skilled and unstoppable their attackers are, but for all we know their software was terrible and a bored 13-year-old shut down their factory because 13-year-olds do terrible things for no reason because the parts of their brains that let them tell good ideas from bad ideas haven't grown in yet.
The guy responsible for the shit software isn't going to tell the CTO his software is shit, the CTO isn't going to tell the CEO his department is incompetent and needs a good house cleaning, starting from the top, and the CEO isn't going to admit culpability to the insurance companies and shareholders who are ultimately on the hook for the damages.
Actually the Trans-Siberian pipeline made this acceptable, which was a cyber attack in peacetime responsible for the largest man-made non-nuclear explosion in history. Or the Turkish pipeline attack. Or the Enigma Machine hack.
The crucial parts of warfare systems are C4ISR: Command, Control, Communications, Computation, Intelligence, Surveillance, and Recognizance.
Computer systems have been a target of covert ops for as long as they have existed. What's happening now is that middle-weight nations (North Korea, Iran) and non-state actors (Anonymous, al-Qassam) are now able to get in on the game, which is disrupting the status quo established by the USA and USSR.
[+] [-] joshvm|11 years ago|reply
Factories like this one probably operate at around 60%+ capacity, so they'll be operating sometimes all day, sometimes all night. If you ever get the chance to visit, do so, even if you don't really care about how steel is made. The sheer scale of everything is amazing.
Everything is very big, very hot and if you have to hit the big red button, it costs a lot of money. Unscheduled downtime is very expensive. Steel tends to be workable when it's hot/molten and therefore pliable. If you suddenly stop a machine then you're left with solid steel in places you don't want it which takes a lot of time and effort to remove.
One of the common reactions to this story is "Why didn't they hit the emergency stop?" - the answer is because it costs an absolute fortune to do so.
[+] [-] fennecfoxen|11 years ago|reply
[+] [-] Fuxy|11 years ago|reply
Shouldn't these things be on a separate network protected by an air gap all the time?
Having stuff that doesn't need internet access connected to the internet is like asking for trouble.
[+] [-] rab_oof|11 years ago|reply
[+] [-] sqeezy|11 years ago|reply
but just some facts from my world :-)
first those plants are build for lifespans of over 30 years. general problem is 15(normal review time) years ago no one was thinking about network security as we thinking about it know . most businesses didn't even have a large internal network wich did include the production and were connected to the internet.
second you can't just shutdown this things. if you have to shutdown a blast furnace we are talking about minimum stand time of 5-7 days. calculate about 400k to 1m € per day on standstill cost. and that is only for the blast furnace. if the blast furnace is not running in some steel plant NOTHING will run. (e.G. hot rolling plants)
third there is no good solution on the market. if some of your guys would look into the software wich is sometimes running those large machines you would get sick to your stomach. As a more security focused person in my plant just to convince management to change the std admin passwords was a handful (well that changed like a year or two years ago). The thing is market decides what security is gonna be implemented. since there has now been a breach and a very expensive one most companies i am talking to are more focused on security now. The thing is they won't just throw away their software stack they worked on for 30 years. and reviewing software is hard and time consuming. so it will be interesting how this is developing.
and no i am not working in that plant ... :-)
and sorry for the bad english
[+] [-] nightcracker|11 years ago|reply
[+] [-] ArchD|11 years ago|reply
And, if they need to use the Internet on-site, can't they make an air gap and segregate the computers that can access the Internet from computers that can access the plant machinery?
[+] [-] HCIdivision17|11 years ago|reply
This is where I begin ranting on the topic.
Why? Because plants aren't secure. They're meant to run all the time by people who may or may not know how to properly use a mouse. There will be passwords taped to monitors, systems that automatically log in to prevent start up delays, and authentication of the order that it-better-just-work-by-default. Under no circumstances should that damn system ever, ever be exposed to the outside world. Not by Ethernet, wireless, or flash drive. It's a young innocent facing the cruel, brutal internet; it's going to get hurt.
New plants are fancy and have highly trained workers with brilliant industrial IP wireless systems with state of the art VM servers and oh god did that guy just plug his phone charger into the damn wrapper HMI and now Windows media player has popped up and no one can acknowledge alarms (this partly why PCs tend to be in large metal boxes, that and dirt/water). No imagine that sort of silliness, but driven by the less savory from outside the intranet.
Just don't risk it. You'll never have a problem, and no one is ever going to care enough to hurt the plant... Except but for that one time when suddenly all the convenience and hubris won't bring back the machine that just slagged itself from some malicious command from outside.
[+] [-] superuser2|11 years ago|reply
As it becomes more clear that yes, someone will go to that trouble and it will have catastrophic consequences, you would hope that these things would get better.
More likely, someone will pay a "security consulting" company $100 million to run a Nessus scan and tell them to turn on Automatic Updates on their Windows infrastructure.
[+] [-] HCIdivision17|11 years ago|reply
If you need to get on the process network, use a VPN, and only open to a machine that can't actually run equipment. A programming terminal may be made available to save costs so an integrator doesn't need to fly in for every support call, but these access points tend to require a VPN through at least two firewalls. (And even then, often you would still insist on them coming in person, for all manner of other reasons.)
[+] [-] ArchD|11 years ago|reply
[+] [-] mweatherill|11 years ago|reply
[+] [-] Animats|11 years ago|reply
Interestingly, there was a cooling water leak and an emergency shutdown at a steel plant in Pakistan in October. That plant is still off line. That's probably unrelated, though.
http://www.newspakistan.pk/2014/10/27/pakistan-steel-mills-r...
[+] [-] frik|11 years ago|reply
Steel plants run for years without a shut down, so this was a large scale incident as the had to shut it down because of major damage.
Not related to the plant in Germany in any way, just to get you an idea how some other steel plants operate: C# WinForm based GUI control room app and Java based server app on Windows server. The server controls the various SPS. Several steel plants around the world were build with that software setup and it was not designed to be connected to the internet.
[+] [-] spacecowboy_lon|11 years ago|reply
[+] [-] dang|11 years ago|reply
Edit: and also from http://arstechnica.com/security/2014/12/computer-intrusion-i..., which points to this.
[+] [-] machrider|11 years ago|reply
[+] [-] afarrell|11 years ago|reply
[+] [-] rebootthesystem|11 years ago|reply
"describing the technical skills of the attacker as “very advanced.”"
And
"not only was there evidence of a strong knowledge of IT security but also extended know-how of the industrial control and production process."
And HN rushes to judgement to quickly blame workers who can't use a mouse and Microsoft.
Yes, the average worker in a manufacturing plant is not a CS grad. It is the job of engineers to develop systems that are usable by, well, the target user.
Most Heart Surgeons don't have a CS degree. And based on meeting a number of them during the course of my business I am comfortable saying that quite a few of them are "computer challenged". Yet, most of us would not have a problem being on that operating table, yes, with a room full of computers, a good number of them running MS software and with an OR team that is likely to use the same "123456" password on everything.
In a hospital you have IT and engineers who setup an infrastructure medical professionals can use. The same is true of steel plants. Yes, there's probably a lot more older code in your average steel plant. I just don't think characterizing them as IT or security morons migt be fair.
The BSI characterized the attackers as sophisticated across disciplines. Let's not engage in senseless conjecture.
I've owned and operated a small manufacturing plant consisting mostly of what I call "big iron" CNC equipment. Things are seldom as simple as discussions on various fora on the 'net would like them to be. Yes, in my case I air-gapped the plant and even individual machines and remote monitoring was done through a separate network that had no command-and-control capabilities at all, just sensing and reporting. There was no way to jump from the sensing network to command-and-control of any one machine, much less the plant. Even if you were physically at the factory this was pretty much impossible. Nobody wants a CNC milling machine with a 30HP spindle controllable from the internet. People are not that stupid...even if they can't use a mouse.
[+] [-] DaveSapien|11 years ago|reply
[+] [-] ars|11 years ago|reply
People do not work that hard to destroy something without a reason. Someone was really mad at them - ex employee maybe?
[+] [-] nisa|11 years ago|reply
There is also this: http://www.heise.de/security/meldung/Verwundbare-Industriean... (In german)
[+] [-] saalweachter|11 years ago|reply
The companies attacked always go on about how skilled and unstoppable their attackers are, but for all we know their software was terrible and a bored 13-year-old shut down their factory because 13-year-olds do terrible things for no reason because the parts of their brains that let them tell good ideas from bad ideas haven't grown in yet.
The guy responsible for the shit software isn't going to tell the CTO his software is shit, the CTO isn't going to tell the CEO his department is incompetent and needs a good house cleaning, starting from the top, and the CEO isn't going to admit culpability to the insurance companies and shareholders who are ultimately on the hook for the damages.
[+] [-] mokash|11 years ago|reply
[+] [-] higherpurpose|11 years ago|reply
[+] [-] s_q_b|11 years ago|reply
The crucial parts of warfare systems are C4ISR: Command, Control, Communications, Computation, Intelligence, Surveillance, and Recognizance.
Computer systems have been a target of covert ops for as long as they have existed. What's happening now is that middle-weight nations (North Korea, Iran) and non-state actors (Anonymous, al-Qassam) are now able to get in on the game, which is disrupting the status quo established by the USA and USSR.
[+] [-] warble|11 years ago|reply
[+] [-] unknown|11 years ago|reply
[deleted]