Staged Hack Causes Generator to Self-Destruct

An anonymous reader writes "It has been revealed that in a U.S. Department of Homeland Security exercise codenamed 'Aurora' conducted in March of this year, researchers were able to cause a power generator to self-destruct remotely via a hack which changed the operating cycle of the generator. 'Government sources said changes are being made to both computer software and physical hardware to protect power generating equipment. And the Nuclear Regulatory Commission said it is conducting inspections to ensure all nuclear plants have made the fix. Industry experts also said the experiment shows large electric systems are vulnerable in ways not previously demonstrated.'"

this should not be possible

because the automation system controlling the infrastructure is not connected to a public network, like say, the internet - right ?
If it is, then someone should probably do some quick patching asap.

Re:this should not be possible

You see they want remote control and monitoring but they also don't want to be on the Internet. They would have to build their own network, unless they are NSA, FBI or AT&T they cannot do that easily. Even then, once there is any remote control, the attacker doesn't have to jump over the fence of the power station, they have a choice to break one window of the building where the point of remote control is.

Re:this should not be possible

Our company has all our generators (and many other things) remotely controlled, and none of those systems are available to the public internet. We have it all captive on our own infrastructure.
The local power utility ( I know several of their techs who work on the telemetry gear) also has a remote control system which in entirely on their own infrastructure, and has no interconnection with any system that is accessible from a public network.
It may not be the absolutely cheapest way to do things, but it's also a lot more secure.
What's the cost of this sort of failure compared to doing it "right" in the first place?

Re:this should not be possible

I'd guess most people here have never read about power grid synchronization. Unless your power grid is DC isolated, it shares data telemetry data with other systems in the grid. Any one of these systems getting hacked can put the entire network at risk. There are many ways to damage a generator if you understand what causes it to trip from the system. Delaying the disconnect from the power grid, for even a short amount of time can cause substantial damage.

http://groups.google.com/group/alt.engineering.electrical/browse_thread/thread/c6a2399745b5413a/dcdf9906b70b85b1%23dcdf9906b70b85b1 [google.com]
http://www.google.com/search?hl=en&q=power+grid+synchronization+failure&btnG=Search [google.com]

Re:this should not be possible

They would have to build their own network, unless they are NSA, FBI or AT&T they cannot do that easily.

What the hell is happening to /.? Has NOBODY here ever heard of a LEASED LINE?

Call up Verizon or AT&T, tell them you want a T1 from point A to point B. You pay them a few dollars every month, and you have a direct, and fully-private connection from A to B.

Public networks aren't the only way to communicate.

Re:this should not be possible

It is. It has to be. It would be ideal if you could run isolated networks, but it's impractical. Let's say you run a facility with some gas turbine generators, as in this example. The generator package has to communicate with the control system. The control system has to communicate with the "business" network (for record-keeping, among other reasons), and the business network has to be connected to the internet. There are lots of things you can do to help secure the various levels of the network, e.g. firewalls, vLANs, packet filtering and inspection, intrusion detection and response, etc., but there still is a data path going all the way out from the lowest levels out to the "real world".

(Our company has also been working with Idaho National Labs on this exact issue, can you tell? The government is taking it pretty seriously)

There are a few problems. For example, there's a lot of old control gear out there, and if it talks ethernet, it assumes that anything it receives is legitimate. Also, the equipment involved is produced in small enough quantities that there can't be a great deal of effort expended on security features. It's not like Windows, where millions and millions of copies are sold, and lots of people actively look for holes.

Re:this should not be possible

.....has to communicate .....

Really, has to? Electric systems have been around since the days of Edison and worked just fine without networks, specifically the Internet. Sacrificing security for convenience is a bad idea that Microsoft has amply demonstrated. Why can a power plant not be controlled locally, by a human operator, like they were in the past. Remote reading is a lot different than remote control. Much of this remote control pressure comes from bean counters in management. They want to eliminate the cost of hiring workers wherever possible.

Normally, each generator, transformer and other equipment has safety devices that shut the machine down BEFORE any damage happens. Whatever happened to those? Do they depend on computers for that safety function now, that a simple relay or circuit breaker used to provide? If the setup in that experiment corresponds to the way power systems are run today, perhaps it's time to take a step into the past.

Re:this should not be possible

because the automation system controlling the infrastructure is not connected to a public network, like say, the internet - right ?

You know, the internet isn't the only network out there. The telephone system is another, with wetware acting as clients and servers. For example:

JOE (technician): *rrring*.. hello?
JACK (mischievous social engineer): Hey Joe, this is Terry at central control
JOE: Hi Terry, what can I do for you?
JACK: I need you to offset the timing on the third generator coil by 20% please.
JOE: Uh? 20%? That sounds dangerous.
JACK: It's urgent! the power-grid is not stable, if you don't do this, we'll have New York in the dark!
JOE: erh.. I really need to talk to my supervisor for this. Who did you say you were?
JACK: I've already talked to your supervisor. John's gonna be really pissed off if you don't do this!
JOE: Well ok then. Here goes...
**KABOOM**

See? no need for any internet, wetware can be hacked too.

Not possible

As someone who as worked in this position in a power station, let me say that this social engineering attack is not likely. You very quickly learn the names, attitudes, and voices of all the people that frequently call asking for changes to the generators. The number of people calling for these changes is usually a handful, 5 or less. If someone odd calls, we would often ask if another guy we knew was on vacation or sick.

If someone we never had heard of called asking for something strange, I would have definitely asked to talk to someone I knew at the independent system operator, emergency or not.

Don't connect it up

There is a really simple and quick fix for this problem - don't connect the control equipment to a (public) computer network.

What is more interesting than the fact this was possible is the fact that some numb skull thought it might be a good idea to link critical control systems to a public network. I can see that there is scope for remote control, especially with a nuclear plant, but I hardly think sending the data over the Intertubes is the correct way to do it.

Re:Don't connect it up

Except that would never work with how the power grid is setup. The plants all communicate with Central Control. (I know because I happen to work for an Electric Company) Central Control is a big room with video walls the likes of which you have never seen! (Our main one happens to be the largest video wall in North America) These control centers are (gues what!) controlling how much power goes out across the lines at any given moment. And it has to be carefully controlled otherwise you get a sag or a spike which does all sorts of damage.

In addition to the Central Control there are Regional Dispatch Offices which have information about the grid as well. These mainly coordinate repair and upgrade efforts. But, they need to know which circuits are hot because people's lives are on the line.

So, simply isolating the plants would not work. Certainly not in our day and age.

Why mention Nuclear?

I don't understand why Nuclear power needed to be singled out. The electrical generators are pretty similar regardless of the fuel source. And if it blows up, it's not going to take the nuclear reactor / coal furnace / (insert steam source here) with it, since they tend to be very well separated from each other.

Re:Why mention Nuclear?

The parent post is profoundly ignorant of how a modern nuclear reactor works.

Um, WHY was the generator on the internet?!!

I'm no computer security expert but I do know of the world's most unhackable firewall -- it's called a one inch air gap. Put that gap between the network cable and the NIC and nobody is gaining access.

Yes, I know power plants will require some net access for web, email, etc. But the office worker network and the command and control computers and network for the generators should have nothing to do with each other! Separate systems, no network connectivity, the plant software should be operating in a vacuum bubble. The rest of the world should not exist for it, no way, no how. Oh, need to install a patch for the software? After being thoroughly tested and vetted on a proofing system, the software is then installed the old-fashioned way, off of CD-ROM's. Now if someone can fuck with the CD-ROM's, THAT I can understand. I can buy the plausibility of the NSA printer hack [vmyths.com], even if it was a hoax. (NSA puts a virus on printers heading to Iraq, takes down their network.) The story about the CIA sabotaging software for equipment the Russians were buying to use in their pipelines [damninteresting.com] is true. These are secure systems completely cut off from external contact that were sabotaged by the insertion of compromised components that were not detected. That makes perfect sense.

It always bothers me when I see movies showing hackers getting in to some place and gaining access to files on servers that should never have a connection to the outside world. Then again, maybe I'm giving the fictional syadmins of the target systems too much credit. Who knows, maybe next week we'll read about some Korean hackers who were able to compromise a Minuteman silo and add it to their botnet.

Re:Um, WHY was the generator on the internet?!!

You've also got to remember, all it takes is one employee with a grudge, or who you aren't paying enough, and all the air gaps in the world won't help you. There is never a single solution.

Fire employees, turn off computers. I'm feeling grumpy.

They are connected to the Internet

I am the system administrator for a large state government agency. Recently I was essentially forced to connect a Windows XP boiler control system for an electrical generation plant to the Internet, so that the vendor can do remote maintenance. If I hadn't found out about it, it would be connected directly without even a firewall... This system had no anti-virus software, and of course it has a popular remote-control software installed for the vendor's access. The only reason I can sleep at night is that the plant is far away from any populated area, and may be shut down due to other reasons soon. I will be sending this video to a number of people in an email today.

Disconnecting is NOT an option

These post are getting ridiculous. Too many people are saying "why don't they just disconnect it from the network?" and getting modded as "insightful".

It's NOT that simple! If they are connected to the network, there is probably a very good reason for it, and not just cause some engineer wants to check his email and download pr0n while listening to the generators hum.

These generators more than likely are controlled by self-optimizing systems based on a variety of data that is collected. If they're providing power to various remote sites, they need the internet for gathering data from those sites.

The internet is more than just a public free-for-all, it is the communication medium for many business/mission-critical systems (see LehiNephi's response above). They really just need to have the right security in place to keep it safe.

Re:Disconnecting is NOT an option

s/the internet/a private wan

why do you need internet (the public one, with no QoS) to have remote access from one point (data collecting / stat computer) to the power plant ?

Yes, the data have to be collected from somewhere, but why not make a private WAN (or a VPN if best-effort QoS is OK for you) for this ? It's not about playing WoW with your neighbour, it's about remote controlling a nuclear core, so maybe it would make sense.

Re:Disconnecting is NOT an option

If they are connected to the network, there is probably a very good reason for it...

Lazyness? Insanely stupid cost cutting?

Yes, the components of the system need to get data back to the dispatcher, and receive instructions in return. No, that doesn't require the internet. You can use a modem on a leased line. Yes, it really is possible to send and receive data without the intarweb.

The internet is a cheap, insecure way to accomplish what should be done on an expensive, secure, private network.

There are Easier Ways...

There are easier ways to damage the bulk power grid (or local transmission). Pick up a rifle at your nearest sporting goods store. Go to your nearest transmission substation (or even large generating plant). Take a shot at the porcelain on one of the transformer bushings. Kablam! You just removed a few hundred MW (or perhaps more) or generating capacity or transfer capability and caused millions of dollars in damage. If it's a generating station, the cost of lost revenue could drive the total to 70 or 80 million. Actually, I have seen bushings with bullet holes. Obviously not that common, or something would be done about it, but it does happen. It won't always cause an immediate and catastrophic failure, but it certainly can. Especially if one keeps trying... The bigger danger to this nations power grid is lack of investment and a severe brain drain in engineering personnel.

Jumping Generators

What a bunch of sad geeks we've become. Instead of crying about how it was connected to the 'net i watched the video.

I'd like to know what they did to make a multi-ton generator JUMP like that thing did. After a few jumps there were a couple chunks of black stuff flying around. If you watch the "full" video it's clear they cut it at least once if not more. I'm guessing it took them quite a long while to get the generator to "blow up".

Anyone have thoughts as to how they did it? I'm going to guess they messed with the fuel/air mix or delivery and caused a massive backfire while under/overloading the alternator side. I'd guess for kicks they also forcibly turned off the cooling fans creating an over-temp in the engine. Assuming i'm right and they cut out 95% of the video length that explains it a bit better. The failure seemed two-fold: A failed main-crankshaft seal spewed out white "smoke" (read over-temp coolant) and something up by the valves making black smoke.

This is probably something you could do to a regular car if you were poking around in the engine management computer.

The threat is real

We know that, because *we* did it to the Soviets. http://www.msnbc.msn.com/id/4394002 [msn.com]

And their machines weren't even connected to the internet. So all the people who are saying, "Just disconnect it", well, that's not good enough. We have to engineer systems that are hardened and handle failure gracefully. And don't use stolen software.

It is mostly bunk

There is no such thing as an "operating cycle" to change for a generator.

The generator pictured in the video is not the kind used in large power plants. It appears to be a diesel generator similar to the kind that is used for backup power in many buildings. Backup generators are typically 1 MW or lesss, whereas big power plant generators are 1000 MW or more. It is like comparing a RC controlled model airplane with a 747. Besides being bigger, the 747 and the power plant will have much more elaborate systems to protect things from damage and destruction caused by malfunctioning equipment and/or misbehaving control systems. When there are billions of dollars and /or human lives at stake, one invests more in safeguards such as electromechanical relays, breakers and other non digital gadgets.

The thing that could cause the generator to jump and destroy itself like in the video is to attempt to synchronize it with the grid out of phase or at the wrong speed. Another post in this thread, "This has happened before computer controls" by Maximum Prophet hit on the correct answer. In small, unattended, backup generators synchronization may be automated by computer, but in large power plants nobody trusts the computer enough to allow this critical operation to be automated. It is still typically done by hand with the aid of old fashioned non-digital equipment. Even if one did mis-synchronize a generator (and it does happen) other protective devices shut things down quickly to limit the scope of damage. And yes, mis-synchronization does happen in real life every once in a while, usually in a brand new installation and usually because the instruments are wired up wrong. The result can be damage sometimes, but I never heard of it destroying a whole plant.

That is not to say that cyberwar is not a threat, nor to say that it is not good policy to isolate all critical control computer from the net. Again its a matter of money. If you are running a $5 billion power plant, your budget is big enough to hire real people to come and maintain systems rather than using remote diagnostics. Or, if you do want remote diagnostics, you can afford to use leased private lines rather than the internet. Power plants and the power grid can afford gold standard security and they should be required to do it. I don't oppose the security thrust, but I do oppose the hyped up scare tactics designed to panic us into unwise government spending.

I spent most of my life modeling power plants and their control systems to build operator training simulators. As part of training, we inject myriads of simulated malfunctions. As part of debugging of the models, we get to see just about every detail of the plant and its control and its safeguards working incorrectly before we debug them and make them correct. That gave me and others experiences up to our chinny chin chins about what can go wrong and what the consequences might be.

I'm afraid that what this is about is another naked grab for government money and using scare tactics to get it. Mr. Joe Weiss in the video works for EPRI. He, and the government committee on critical infrastructure protection, were both singing the song in 1999 that no matter what Y2K bugs might exist, they couldn't do any real harm. Get it? Not that the Y2K bugs didn't exist or would be fixed (at proved to be the case) but that they couldn't do any substantial harm no matter what. Now these same people are saying that a few hacks can cause widespread and catastrophic damage. One can not argue both sides of this issue and keep credibility. If a control system misbehaves, it matters not whether the problem is inadvertent or malevolent. Yet these people pooh pooh the risk of inadvertent bugs yet hype the danger of malevolent ones. It's bunk.

EPRI wants $100 billion to automate everything in the power grid as a massive research project. Next they'll want another $250 billion to secure it from cyberwar threats. DOE wants a national DOE control center for the

Re:It is mostly bunk

whereas big power plant generators are 1000 MW or more.

Heh... that's it? I once heard of a professor somewhere that was able to build a portable generator, small enough to fit in the rear half of a small car, capable of outputting 1,210 MW...

Re:Bruce Willis will prevent this from ever happen

Did anyone else immediately think of Live Free or Die Hard when reading this?

No, because you're the only one who watched that movie.