Staged Hack Causes Generator to Self-Destruct

An anonymous reader writes "It has been revealed that in a U.S. Department of Homeland Security exercise codenamed 'Aurora' conducted in March of this year, researchers were able to cause a power generator to self-destruct remotely via a hack which changed the operating cycle of the generator. 'Government sources said changes are being made to both computer software and physical hardware to protect power generating equipment. And the Nuclear Regulatory Commission said it is conducting inspections to ensure all nuclear plants have made the fix. Industry experts also said the experiment shows large electric systems are vulnerable in ways not previously demonstrated.'"

this should not be possible

[arabagast (462679)]

because the automation system controlling the infrastructure is not connected to a public network, like say, the internet - right ?
If it is, then someone should probably do some quick patching asap.

Re:this should not be possible

[drgonzo59 (747139)]

You see they want remote control and monitoring but they also don't want to be on the Internet. They would have to build their own network, unless they are NSA, FBI or AT&T they cannot do that easily. Even then, once there is any remote control, the attacker doesn't have to jump over the fence of the power station, they have a choice to break one window of the building where the point of remote control is.

Re:

[SQLGuru (980662)]

Anyone wonder why they've been researching Ethernet over Powerlines? They already have the cables deployed all over the place, they just need to get the data flowing along with all of the other electrons.

Layne

Re:

[Hijacked Public (999535)]

If you mean BPL [howstuffworks.com], it has been rolled out in a few rural areas of the US. I know Cinergy has a lot of BPL customers around Cincinatti.

I hoping it come to southern Indiana soon. Fast up and down.

Re:this should not be possible

[kent_eh (543303)]

Our company has all our generators (and many other things) remotely controlled, and none of those systems are available to the public internet. We have it all captive on our own infrastructure.
The local power utility ( I know several of their techs who work on the telemetry gear) also has a remote control system which in entirely on their own infrastructure, and has no interconnection with any system that is accessible from a public network.
It may not be the absolutely cheapest way to do things, but it's also a lot more secure.
What's the cost of this sort of failure compared to doing it "right" in the first place?

Re:this should not be possible

[PlusFiveTroll (754249)]

I'd guess most people here have never read about power grid synchronization. Unless your power grid is DC isolated, it shares data telemetry data with other systems in the grid. Any one of these systems getting hacked can put the entire network at risk. There are many ways to damage a generator if you understand what causes it to trip from the system. Delaying the disconnect from the power grid, for even a short amount of time can cause substantial damage.

http://groups.google.com/group/alt.engineering.electrical/browse_thread/thread/c6a2399745b5413a/dcdf9906b70b85b1%23dcdf9906b70b85b1 [google.com]
http://www.google.com/search?hl=en&q=power+grid+synchronization+failure&btnG=Search [google.com]

Re:

[legirons (809082)]

You mean they need to jump a motorcycle into the guard post from an adjacent building, break into the control centre, and run nmap on a terminal on the internal network?

Re:this should not be possible

[evilviper (135110)]

They would have to build their own network, unless they are NSA, FBI or AT&T they cannot do that easily.

What the hell is happening to /.? Has NOBODY here ever heard of a LEASED LINE?

Call up Verizon or AT&T, tell them you want a T1 from point A to point B. You pay them a few dollars every month, and you have a direct, and fully-private connection from A to B.

Public networks aren't the only way to communicate.

Re:this should not be possible

[LehiNephi (695428)]

It is. It has to be. It would be ideal if you could run isolated networks, but it's impractical. Let's say you run a facility with some gas turbine generators, as in this example. The generator package has to communicate with the control system. The control system has to communicate with the "business" network (for record-keeping, among other reasons), and the business network has to be connected to the internet. There are lots of things you can do to help secure the various levels of the network, e.g. firewalls, vLANs, packet filtering and inspection, intrusion detection and response, etc., but there still is a data path going all the way out from the lowest levels out to the "real world".

(Our company has also been working with Idaho National Labs on this exact issue, can you tell? The government is taking it pretty seriously)

There are a few problems. For example, there's a lot of old control gear out there, and if it talks ethernet, it assumes that anything it receives is legitimate. Also, the equipment involved is produced in small enough quantities that there can't be a great deal of effort expended on security features. It's not like Windows, where millions and millions of copies are sold, and lots of people actively look for holes.

Re:

[jimstapleton (999106)]

It seems changes should be manageable on-site, while offsite monitoring should be done by dumps.

i.e. You could burn discs with the necessary logs/data, you could set up a send-only piece of hardware, etc.

Re:this should not be possible

[arminw (717974)]

.....has to communicate .....

Really, has to? Electric systems have been around since the days of Edison and worked just fine without networks, specifically the Internet. Sacrificing security for convenience is a bad idea that Microsoft has amply demonstrated. Why can a power plant not be controlled locally, by a human operator, like they were in the past. Remote reading is a lot different than remote control. Much of this remote control pressure comes from bean counters in management. They want to eliminate the cost of hiring workers wherever possible.

Normally, each generator, transformer and other equipment has safety devices that shut the machine down BEFORE any damage happens. Whatever happened to those? Do they depend on computers for that safety function now, that a simple relay or circuit breaker used to provide? If the setup in that experiment corresponds to the way power systems are run today, perhaps it's time to take a step into the past.

Re:

[Hatta (162192)]

The control system has to communicate with the "business" network (for record-keeping, among other reasons)

Use Sneakernet, not Ethernet.

Re:

[russotto (537200)]

The problem is in allowing any remote control of the system, which the utility wants to happen so that a central facility can control any generator. Here, we have four power generation facilities, all of which are managed from a central control at the utilities main office downtown. They choose to use the internet to make those connections, because it's MUCH cheaper than stringing dedicated data lines from the generation plants to the central office.

I'm pretty sure that's a false dilemma. Doesn't the ph

Re:

[LeRandy (937290)]

At least here in the UK, Telemetry and control signals are carried over the National Grid itself, nowadays using an optic fibre that runs alongside the earth wire. Case Study [teligenceuk.com].

I see no reason why all telemetry and control signals should not be carried in narrow- or broadband communications along the power infrastructure itself, and then restricted to a physically separate infrastructure when being processed. Data links to business systems can be provided using a one-way connection (Serial or optical). If yo

Re:this should not be possible

[Rosco P. Coltrane (209368)]

because the automation system controlling the infrastructure is not connected to a public network, like say, the internet - right ?

You know, the internet isn't the only network out there. The telephone system is another, with wetware acting as clients and servers. For example:

JOE (technician): *rrring*.. hello?
JACK (mischievous social engineer): Hey Joe, this is Terry at central control
JOE: Hi Terry, what can I do for you?
JACK: I need you to offset the timing on the third generator coil by 20% please.
JOE: Uh? 20%? That sounds dangerous.
JACK: It's urgent! the power-grid is not stable, if you don't do this, we'll have New York in the dark!
JOE: erh.. I really need to talk to my supervisor for this. Who did you say you were?
JACK: I've already talked to your supervisor. John's gonna be really pissed off if you don't do this!
JOE: Well ok then. Here goes...
**KABOOM**

See? no need for any internet, wetware can be hacked too.

Not possible

[dj245 (732906)]

As someone who as worked in this position in a power station, let me say that this social engineering attack is not likely. You very quickly learn the names, attitudes, and voices of all the people that frequently call asking for changes to the generators. The number of people calling for these changes is usually a handful, 5 or less. If someone odd calls, we would often ask if another guy we knew was on vacation or sick.

If someone we never had heard of called asking for something strange, I would have definitely asked to talk to someone I knew at the independent system operator, emergency or not.

Re:

[arivanov (12034)]

IIRC, The US network is connected in places or separated by weak bastion hosts. If you do not remember the case when Slammer caused blackouts in the North East, some of us do.

Even if the USA network was not connected, the control systems themselves use laughable authentication (if any). Most other networks are similar. They have been built by control automation engineers whose knowledge of networking and security is somewhere between laughing stock and none. This is valid for the rest of the world, not just

Re:

[Rosco P. Coltrane (209368)]

Yeah, if only the planes during 9/11 weren't connected to a public network, like the internet... Oh, wait. They weren't.

Yes they were. If you ever placed a 20$/min phone call from a plane, you would know.

But I digress, telephones obviously didn't cause the planes to crash.

Don't connect it up

[squoozer (730327)]

There is a really simple and quick fix for this problem - don't connect the control equipment to a (public) computer network.

What is more interesting than the fact this was possible is the fact that some numb skull thought it might be a good idea to link critical control systems to a public network. I can see that there is scope for remote control, especially with a nuclear plant, but I hardly think sending the data over the Intertubes is the correct way to do it.

Re:

[LehiNephi (695428)]

There's one problem with that: in today's world, data has to flow back to headquarters. Take an oil production facility for example. The plant has to send back a daily report detailing exactly how much gas/oil/water/CO2/H2S/sand/whatever is produced. Gas turbines send data back to the manufacturer for performance evaluation, maintenance scheduling, and troubleshooting. Yes, someone could do it manually, but there are myriad other functions that require network connectivity beyond the control system.

Re:Don't connect it up

[theotherbastard (939373)]

Except that would never work with how the power grid is setup. The plants all communicate with Central Control. (I know because I happen to work for an Electric Company) Central Control is a big room with video walls the likes of which you have never seen! (Our main one happens to be the largest video wall in North America) These control centers are (gues what!) controlling how much power goes out across the lines at any given moment. And it has to be carefully controlled otherwise you get a sag or a spike which does all sorts of damage.

In addition to the Central Control there are Regional Dispatch Offices which have information about the grid as well. These mainly coordinate repair and upgrade efforts. But, they need to know which circuits are hot because people's lives are on the line.

So, simply isolating the plants would not work. Certainly not in our day and age.

Why mention Nuclear?

[brucmack (572780)]

I don't understand why Nuclear power needed to be singled out. The electrical generators are pretty similar regardless of the fuel source. And if it blows up, it's not going to take the nuclear reactor / coal furnace / (insert steam source here) with it, since they tend to be very well separated from each other.

Re:Why mention Nuclear?

[by Anonymous Coward]

The parent post is profoundly ignorant of how a modern nuclear reactor works.

Remotely caused power generator to self-destruct?

[permaculture (567540)]

"cause a power generator to self-destruct remotely". This seems unlikely.

What probably happened was that they "remotely caused a power generator to self-destruct."

/stickler

Re:Remotely caused power generator to self-destruc

[morgan_greywolf (835522)]

Hi! This is Chief Rufus Xavier Sarsaparilla of the Grammar Police. Where do we send your check, Lt. Permaculture?

Um, WHY was the generator on the internet?!!

[jollyreaper (513215)]

I'm no computer security expert but I do know of the world's most unhackable firewall -- it's called a one inch air gap. Put that gap between the network cable and the NIC and nobody is gaining access.

Yes, I know power plants will require some net access for web, email, etc. But the office worker network and the command and control computers and network for the generators should have nothing to do with each other! Separate systems, no network connectivity, the plant software should be operating in a vacuum bubble. The rest of the world should not exist for it, no way, no how. Oh, need to install a patch for the software? After being thoroughly tested and vetted on a proofing system, the software is then installed the old-fashioned way, off of CD-ROM's. Now if someone can fuck with the CD-ROM's, THAT I can understand. I can buy the plausibility of the NSA printer hack [vmyths.com], even if it was a hoax. (NSA puts a virus on printers heading to Iraq, takes down their network.) The story about the CIA sabotaging software for equipment the Russians were buying to use in their pipelines [damninteresting.com] is true. These are secure systems completely cut off from external contact that were sabotaged by the insertion of compromised components that were not detected. That makes perfect sense.

It always bothers me when I see movies showing hackers getting in to some place and gaining access to files on servers that should never have a connection to the outside world. Then again, maybe I'm giving the fictional syadmins of the target systems too much credit. Who knows, maybe next week we'll read about some Korean hackers who were able to compromise a Minuteman silo and add it to their botnet.

Re:

[jamesh (87723)]

I have about 90 inches of air between my computer and the network, and it's not stopping me.

The "1 inch (or mm) air gap" idea is a good one, but getting harder and harder to implement. If a tech has a laptop connected to the internal network, and has wireless enabled, and its in range of the hacker then you have a problem (in theory - see the recent apple wireless compromise)

If he has a PC connected to the internal network with no wireless, but has his phone connected to it via USB, then in theory that coul

Re:Um, WHY was the generator on the internet?!!

[jollyreaper (513215)]

You've also got to remember, all it takes is one employee with a grudge, or who you aren't paying enough, and all the air gaps in the world won't help you. There is never a single solution.

Fire employees, turn off computers. I'm feeling grumpy.

Re:

[Rosco P. Coltrane (209368)]

I'm no computer security expert but I do know of the world's most unhackable firewall -- it's called a one inch air gap. Put that gap between the network cable and the NIC and nobody is gaining access.

Sorry, not enough. Smart hackers up the line voltage in the network cable to 20kV to cross the one inch air gap.

Re:

[SQLGuru (980662)]

Actually, the connections come in when you start looking at feeding operating data into the business processes.....or when you want to monitor the state of the machine to automatically trigger preventative maintenance.....or automated control.

Layne

Decreasing DHS budget...

[bracktra (712808)]

"Fast and resolute mitigating action is needed to avoid a national disaster," the letter said. But five years later, there is no such program. Federal spending on electronic security is projected to increase slightly in the coming fiscal year, but spending in the Department of Homeland Security is projected to decrease to less than $100 million, with only $12 million spent to secure power control systems.

1. Stage PR stunt about an impending 'emergency!!!'.
2. Complain about lack of funding to solve desperate hole in our nation's security.
3. ???
4. Profit!

They are connected to the Internet

[Isbjorn (755227)]

I am the system administrator for a large state government agency. Recently I was essentially forced to connect a Windows XP boiler control system for an electrical generation plant to the Internet, so that the vendor can do remote maintenance. If I hadn't found out about it, it would be connected directly without even a firewall... This system had no anti-virus software, and of course it has a popular remote-control software installed for the vendor's access. The only reason I can sleep at night is that the plant is far away from any populated area, and may be shut down due to other reasons soon. I will be sending this video to a number of people in an email today.

Operating System?

[trelanexiph (605826)]

From TFA "researchers were able to cause a power generator to self-destruct remotely via a hack which changed the operating cycle of the generator"

You mean they upgraded it to Microsoft Windows Vista?

TV is 3 years faster than Government

[suv4x4 (956391)]

The TV movie Category 6: Day of Destruction [imdb.com] went into details that US power plants are vulnerable to remote attacks, and featured such a guy who managed to make generators self-destruct from his home PC (he died when connectivity was cut off, and realising what he did, he went to the power plant to fix things locally, but too late).

And there we go, 3 years later the government wakes up to the threat as well.

Guess my advice to government fellows is: watch more TV, it'll raise your IQ. OMG the irony :(

Disconnecting is NOT an option

[ExE122 (954104)]

These post are getting ridiculous. Too many people are saying "why don't they just disconnect it from the network?" and getting modded as "insightful".

It's NOT that simple! If they are connected to the network, there is probably a very good reason for it, and not just cause some engineer wants to check his email and download pr0n while listening to the generators hum.

These generators more than likely are controlled by self-optimizing systems based on a variety of data that is collected. If they're providing power to various remote sites, they need the internet for gathering data from those sites.

The internet is more than just a public free-for-all, it is the communication medium for many business/mission-critical systems (see LehiNephi's response above). They really just need to have the right security in place to keep it safe.

Re:Disconnecting is NOT an option

[makapuf (412290)]

s/the internet/a private wan

why do you need internet (the public one, with no QoS) to have remote access from one point (data collecting / stat computer) to the power plant ?

Yes, the data have to be collected from somewhere, but why not make a private WAN (or a VPN if best-effort QoS is OK for you) for this ? It's not about playing WoW with your neighbour, it's about remote controlling a nuclear core, so maybe it would make sense.

Re:Disconnecting is NOT an option

[nels_tomlinson (106413)]

If they are connected to the network, there is probably a very good reason for it...

Lazyness? Insanely stupid cost cutting?

Yes, the components of the system need to get data back to the dispatcher, and receive instructions in return. No, that doesn't require the internet. You can use a modem on a leased line. Yes, it really is possible to send and receive data without the intarweb.

The internet is a cheap, insecure way to accomplish what should be done on an expensive, secure, private network.

There are Easier Ways...

[xfmr_expert (853170)]

There are easier ways to damage the bulk power grid (or local transmission). Pick up a rifle at your nearest sporting goods store. Go to your nearest transmission substation (or even large generating plant). Take a shot at the porcelain on one of the transformer bushings. Kablam! You just removed a few hundred MW (or perhaps more) or generating capacity or transfer capability and caused millions of dollars in damage. If it's a generating station, the cost of lost revenue could drive the total to 70 or 80 million. Actually, I have seen bushings with bullet holes. Obviously not that common, or something would be done about it, but it does happen. It won't always cause an immediate and catastrophic failure, but it certainly can. Especially if one keeps trying... The bigger danger to this nations power grid is lack of investment and a severe brain drain in engineering personnel.

Jumping Generators

[torkus (1133985)]

What a bunch of sad geeks we've become. Instead of crying about how it was connected to the 'net i watched the video.

I'd like to know what they did to make a multi-ton generator JUMP like that thing did. After a few jumps there were a couple chunks of black stuff flying around. If you watch the "full" video it's clear they cut it at least once if not more. I'm guessing it took them quite a long while to get the generator to "blow up".

Anyone have thoughts as to how they did it? I'm going to guess they messed with the fuel/air mix or delivery and caused a massive backfire while under/overloading the alternator side. I'd guess for kicks they also forcibly turned off the cooling fans creating an over-temp in the engine. Assuming i'm right and they cut out 95% of the video length that explains it a bit better. The failure seemed two-fold: A failed main-crankshaft seal spewed out white "smoke" (read over-temp coolant) and something up by the valves making black smoke.

This is probably something you could do to a regular car if you were poking around in the engine management computer.

Re:

[trybywrench (584843)]

looks like a thrown rod, maybe they somehow cut off the supply of oil? I don't think the oil pump is usually under any kind of computer control though. ..maybe they over revved the engine and blew a piston that way. Keep the tach red lined long enough and something bad will happen. I don't know about a backfire, wouldn't a backfire cause a stall in the worst case? It looks like something mechanical broke inside the engine (that shudder) and then it slowly ground to a hault.

The threat is real

[Maximum Prophet (716608)]

We know that, because *we* did it to the Soviets. http://www.msnbc.msn.com/id/4394002 [msn.com]

And their machines weren't even connected to the internet. So all the people who are saying, "Just disconnect it", well, that's not good enough. We have to engineer systems that are hardened and handle failure gracefully. And don't use stolen software.

Money

[Detritus (11846)]

As I've said before, it's all about money. There are almost irresistible forces that lead organizations to connect control systems to the Internet. An isolated private internet is extremely expensive and difficult to maintain. It's so much easier, cheaper, and tempting, to plug that cable into the public internet, perhaps with a crappy firewall to provide an illusion of security. Even if an engineer is willing to stick his neck out and say that it's an unacceptable security risk, he isn't being a team player and will be overruled by someone higher up the food chain.

It is mostly bunk

[anorlunda (311253)]

There is no such thing as an "operating cycle" to change for a generator.

The generator pictured in the video is not the kind used in large power plants. It appears to be a diesel generator similar to the kind that is used for backup power in many buildings. Backup generators are typically 1 MW or lesss, whereas big power plant generators are 1000 MW or more. It is like comparing a RC controlled model airplane with a 747. Besides being bigger, the 747 and the power plant will have much more elaborate systems to protect things from damage and destruction caused by malfunctioning equipment and/or misbehaving control systems. When there are billions of dollars and /or human lives at stake, one invests more in safeguards such as electromechanical relays, breakers and other non digital gadgets.

The thing that could cause the generator to jump and destroy itself like in the video is to attempt to synchronize it with the grid out of phase or at the wrong speed. Another post in this thread, "This has happened before computer controls" by Maximum Prophet hit on the correct answer. In small, unattended, backup generators synchronization may be automated by computer, but in large power plants nobody trusts the computer enough to allow this critical operation to be automated. It is still typically done by hand with the aid of old fashioned non-digital equipment. Even if one did mis-synchronize a generator (and it does happen) other protective devices shut things down quickly to limit the scope of damage. And yes, mis-synchronization does happen in real life every once in a while, usually in a brand new installation and usually because the instruments are wired up wrong. The result can be damage sometimes, but I never heard of it destroying a whole plant.

That is not to say that cyberwar is not a threat, nor to say that it is not good policy to isolate all critical control computer from the net. Again its a matter of money. If you are running a $5 billion power plant, your budget is big enough to hire real people to come and maintain systems rather than using remote diagnostics. Or, if you do want remote diagnostics, you can afford to use leased private lines rather than the internet. Power plants and the power grid can afford gold standard security and they should be required to do it. I don't oppose the security thrust, but I do oppose the hyped up scare tactics designed to panic us into unwise government spending.

I spent most of my life modeling power plants and their control systems to build operator training simulators. As part of training, we inject myriads of simulated malfunctions. As part of debugging of the models, we get to see just about every detail of the plant and its control and its safeguards working incorrectly before we debug them and make them correct. That gave me and others experiences up to our chinny chin chins about what can go wrong and what the consequences might be.

I'm afraid that what this is about is another naked grab for government money and using scare tactics to get it. Mr. Joe Weiss in the video works for EPRI. He, and the government committee on critical infrastructure protection, were both singing the song in 1999 that no matter what Y2K bugs might exist, they couldn't do any real harm. Get it? Not that the Y2K bugs didn't exist or would be fixed (at proved to be the case) but that they couldn't do any substantial harm no matter what. Now these same people are saying that a few hacks can cause widespread and catastrophic damage. One can not argue both sides of this issue and keep credibility. If a control system misbehaves, it matters not whether the problem is inadvertent or malevolent. Yet these people pooh pooh the risk of inadvertent bugs yet hype the danger of malevolent ones. It's bunk.

EPRI wants $100 billion to automate everything in the power grid as a massive research project. Next they'll want another $250 billion to secure it from cyberwar threats. DOE wants a national DOE control center for the

Re:It is mostly bunk

[wawannem (591061)]

whereas big power plant generators are 1000 MW or more.

Heh... that's it? I once heard of a professor somewhere that was able to build a portable generator, small enough to fit in the rear half of a small car, capable of outputting 1,210 MW...

Re:

[cdrguru (88047)]

Nonsense. Do you understand what the output of such a generator would be?

I believe it is very high voltage with not-so-much current. Well over 100,000 volts.

The separation between the output terminals would be larger than the space occupied by the car.

OK, what if I'm wrong and it is lots and lots of current. At 1200MW the output current would require something that isn't going to fit in a car to connect to the output terminals.

Either way, it isn't fitting in the space of a car. Not even an Excursion.

Don't Use The Internet For This

[maz2331 (1104901)]

Whatever the reason's given for connecting any critical infrastructure to the public Internet, it is far too risky of a proposition to seriously consider it. They absolutely should be using private WANs, preferably encrypted eight ways to Sunday.

There is absolutely no excuse whatsoever for making this equipment accessable from the public Internet. None. Zero. Zilch.

Frame Relay T1 lines are cheap nowadays, and they should be using them.

Re:Bruce Willis will prevent this from ever happen

[by Anonymous Coward]

Did anyone else immediately think of Live Free or Die Hard when reading this?

No, because you're the only one who watched that movie.

Re:

[morgan_greywolf (835522)]

Did anyone else immediately think of Live Free or Die Hard when reading this?

No, because you're the only one who watched that movie.

I did....oh wait, did you say that was supposed to be a movie? Gak!

Re:

[Rosco P. Coltrane (209368)]

Like 11 and a half years by now. Name Chernobyl rings a bell?

Hello friend. Now don't panic, but I'm afraid I have to tell you you're stuck in the year 1997.

Re:

[Ajehals (947354)]

I mangled a gear change coming back on a stretch of motorway at about 4am, this was maybe 6 months after passing my test, I'm not entirely sure what I did but it was with a change from 4th to first or 4th to reverse (and yes this was a fairly old car). I must say it was fairly spectacular, the smell of burning clutch, the sparks, the rapid deceleration. But most interestingly when I finally got the car to stop, I found that the clutch was stuck/fused, and I couldn't start the engine at all, 20 minutes late