Staged Hack Causes Generator to Self-Destruct

An anonymous reader writes "It has been revealed that in a U.S. Department of Homeland Security exercise codenamed 'Aurora' conducted in March of this year, researchers were able to cause a power generator to self-destruct remotely via a hack which changed the operating cycle of the generator. 'Government sources said changes are being made to both computer software and physical hardware to protect power generating equipment. And the Nuclear Regulatory Commission said it is conducting inspections to ensure all nuclear plants have made the fix. Industry experts also said the experiment shows large electric systems are vulnerable in ways not previously demonstrated.'"

Re:this should not be possible

[LehiNephi]

It is. It has to be. It would be ideal if you could run isolated networks, but it's impractical. Let's say you run a facility with some gas turbine generators, as in this example. The generator package has to communicate with the control system. The control system has to communicate with the "business" network (for record-keeping, among other reasons), and the business network has to be connected to the internet. There are lots of things you can do to help secure the various levels of the network, e.g. firewalls, vLANs, packet filtering and inspection, intrusion detection and response, etc., but there still is a data path going all the way out from the lowest levels out to the "real world".

(Our company has also been working with Idaho National Labs on this exact issue, can you tell? The government is taking it pretty seriously)

There are a few problems. For example, there's a lot of old control gear out there, and if it talks ethernet, it assumes that anything it receives is legitimate. Also, the equipment involved is produced in small enough quantities that there can't be a great deal of effort expended on security features. It's not like Windows, where millions and millions of copies are sold, and lots of people actively look for holes.

Re:this should not be possible

[SQLGuru]

Anyone wonder why they've been researching Ethernet over Powerlines? They already have the cables deployed all over the place, they just need to get the data flowing along with all of the other electrons.

Layne

They are connected to the Internet

[Isbjorn]

I am the system administrator for a large state government agency. Recently I was essentially forced to connect a Windows XP boiler control system for an electrical generation plant to the Internet, so that the vendor can do remote maintenance. If I hadn't found out about it, it would be connected directly without even a firewall... This system had no anti-virus software, and of course it has a popular remote-control software installed for the vendor's access. The only reason I can sleep at night is that the plant is far away from any populated area, and may be shut down due to other reasons soon. I will be sending this video to a number of people in an email today.

Re:this should not be possible

[arivanov]

IIRC, The US network is connected in places or separated by weak bastion hosts. If you do not remember the case when Slammer caused blackouts in the North East, some of us do.

Even if the USA network was not connected, the control systems themselves use laughable authentication (if any). Most other networks are similar. They have been built by control automation engineers whose knowledge of networking and security is somewhere between laughing stock and none. This is valid for the rest of the world, not just the USA.

I am surprised the control automation allows setting parameters which are outside permitted ranges. This is something control and automation people usually get right. I remember my dad spending months on numerical models of the grid to compile sets of allowed parameters all of which ended being hardcoded in hardware and software. Nothing was left to be adjusted outside these ranges (this was not in the USA though).

One really worrying bit is that this is not USA limited. The same automation software and hardware is used in the UK and quite a few other countries.

Jumping Generators

[torkus]

What a bunch of sad geeks we've become. Instead of crying about how it was connected to the 'net i watched the video.

I'd like to know what they did to make a multi-ton generator JUMP like that thing did. After a few jumps there were a couple chunks of black stuff flying around. If you watch the "full" video it's clear they cut it at least once if not more. I'm guessing it took them quite a long while to get the generator to "blow up".

Anyone have thoughts as to how they did it? I'm going to guess they messed with the fuel/air mix or delivery and caused a massive backfire while under/overloading the alternator side. I'd guess for kicks they also forcibly turned off the cooling fans creating an over-temp in the engine. Assuming i'm right and they cut out 95% of the video length that explains it a bit better. The failure seemed two-fold: A failed main-crankshaft seal spewed out white "smoke" (read over-temp coolant) and something up by the valves making black smoke.

This is probably something you could do to a regular car if you were poking around in the engine management computer.

Re:Don't connect it up

[theotherbastard]

Except that would never work with how the power grid is setup. The plants all communicate with Central Control. (I know because I happen to work for an Electric Company) Central Control is a big room with video walls the likes of which you have never seen! (Our main one happens to be the largest video wall in North America) These control centers are (gues what!) controlling how much power goes out across the lines at any given moment. And it has to be carefully controlled otherwise you get a sag or a spike which does all sorts of damage.

In addition to the Central Control there are Regional Dispatch Offices which have information about the grid as well. These mainly coordinate repair and upgrade efforts. But, they need to know which circuits are hot because people's lives are on the line.

So, simply isolating the plants would not work. Certainly not in our day and age.

I used to work for a SCADA/HMI software vendor

[Anonymous Coward]

I don't usually post anonymously, but I will this time.

I used to be a developer for a SCADA/HMI software vendor. That stands for Supervisory Control And Data Acquisition [wikipedia.org] / Human Machine Interface.

It is quite common for such software to be used in places where its failure could cause injury or death.

Many of our customers put their SCADA systems on the Internet, so that our support staff could work with their systems, as well as to allow our consultant engineers to remotely upload new releases.

One day my boss told me that a lot of our customers didn't use SSL encryption, either because they couldn't be bothered with it, or because they couldn't figure out how to install the server software or certificate correctly.

Anyone with a packet sniffer running on the path between us and our customers could have easily stolen the passwords.

Our product, BTW, ran on Microsoft Windows.

The threat is real

[Maximum Prophet]

We know that, because *we* did it to the Soviets. http://www.msnbc.msn.com/id/4394002 [msn.com]

And their machines weren't even connected to the internet. So all the people who are saying, "Just disconnect it", well, that's not good enough. We have to engineer systems that are hardened and handle failure gracefully. And don't use stolen software.

Re:this should not be possible

[kent_eh]

Our company has all our generators (and many other things) remotely controlled, and none of those systems are available to the public internet. We have it all captive on our own infrastructure.
The local power utility ( I know several of their techs who work on the telemetry gear) also has a remote control system which in entirely on their own infrastructure, and has no interconnection with any system that is accessible from a public network.
It may not be the absolutely cheapest way to do things, but it's also a lot more secure.
What's the cost of this sort of failure compared to doing it "right" in the first place?

Not possible

[dj245]

As someone who as worked in this position in a power station, let me say that this social engineering attack is not likely. You very quickly learn the names, attitudes, and voices of all the people that frequently call asking for changes to the generators. The number of people calling for these changes is usually a handful, 5 or less. If someone odd calls, we would often ask if another guy we knew was on vacation or sick.

If someone we never had heard of called asking for something strange, I would have definitely asked to talk to someone I knew at the independent system operator, emergency or not.

Re:this should not be possible

[PlusFiveTroll]

I'd guess most people here have never read about power grid synchronization. Unless your power grid is DC isolated, it shares data telemetry data with other systems in the grid. Any one of these systems getting hacked can put the entire network at risk. There are many ways to damage a generator if you understand what causes it to trip from the system. Delaying the disconnect from the power grid, for even a short amount of time can cause substantial damage.

http://groups.google.com/group/alt.engineering.electrical/browse_thread/thread/c6a2399745b5413a/dcdf9906b70b85b1%23dcdf9906b70b85b1 [google.com]
http://www.google.com/search?hl=en&q=power+grid+synchronization+failure&btnG=Search [google.com]

Why this was released...

[Kiralan]

My (paranoid?) suspicions are: 1. DHS produced this FUD/PhotoOp (remember, it is CNN) to justify their funding. Their current terrorist prevention accomplishments are in the category of 'See any elephants/terrorists? No? Must mean our elephant/terrorist repellent works' 2. Showing a terrorist target that 'hits closer to home' (no pun intended) for Joe/Jane citizen 3. A reason to let them monitor everything they can on the internet. Their justification would be 'If we see them trying to get in, we can find and stop them.' Seems that a proper firewall / VPN setup would be required/more useful K

Re:Why mention Nuclear?

[Anonymous Coward]

Especially since the largest single energy producing unit is a coal plant out in the midwest called Zimmer, which you can find from EIA files. Admittedly, it just ekes out Palo Verde Nuclear, but its close. Of the 100 largest units in the US, 70 are nuclear units, and 25 are coal. All of them are connected to private SCADA networks linking them to their Local Control Centers. Almost all companies I am aware of have also linked their private networks to their LANs through VPN.

A loss of a generator can cause a frequency disturbance across the whole interconnect, and the grid can only survive about 6 simultaneous losses of that magnitude before you would start tripping underfrequency relays, and companies would start disconnecting from one another and begin dumping load (customers). Each Control Area keeps reserves to recover the loss of their largest unit in 10 minutes, so it has to be quick before the reserve CTs can start. Once blacked out, a CA can reattach to its neighbor for a faster recovery.

The danger is a compromised LAN, leading to compromized VPN authentication, granting remote access to the SCADA network. The cracker would then access several LCCs simultaneously, causing numerous simultaneous trippings across the country, bringing down many CAs. The CAs wouldn't be able to cover their immediate loss, and knocking out a lot of pieces could bring the whole network to its knees, and you wouldn't be able to lean on your neighbor for reserves. The 4 hour duration NorthEast blackout of 2003 caused immense financial damage, so there's no accounting for what a larger outage would do.

Re:I've seen this before.

[Ajehals]

I mangled a gear change coming back on a stretch of motorway at about 4am, this was maybe 6 months after passing my test, I'm not entirely sure what I did but it was with a change from 4th to first or 4th to reverse (and yes this was a fairly old car). I must say it was fairly spectacular, the smell of burning clutch, the sparks, the rapid deceleration. But most interestingly when I finally got the car to stop, I found that the clutch was stuck/fused, and I couldn't start the engine at all, 20 minutes later and I had it started, and moving in second gear at about 20Mph, all the way back home @50 Miles (and yes off the motorway.). Next day, I found that everything worked beautifully, and whereas previously the clutch used to slip quite a bit, had regained a decent bite.

Needless to say it didn't pass its next MOT, but then a £250 C Reg Ford Sierra is something you can drive for a year and then replace.