Please create an account to participate in the Slashdot moderation system

 



Forgot your password?
typodupeerror
×
Security Government

'How 30 Lines of Code Blew Up a 27-Ton Generator' (wired.com) 110

After the U.S. unveiled charges against six members of the Sandworm unit in Russia's military intelligence agency, Wired re-visited "a secret experiment in 2007 proved that hackers could devastate power grid equipment beyond repair — with a file no bigger than a gif." It's an excerpt from the new book SANDWORM: A New Era of Cyberwar and the Hunt for the Kremlin's Most Dangerous Hackers which also remembers the late industrial control systems security pioneer Mike Assante: Among [Sandworm's] acts of cyberwar was an unprecedented attack on Ukraine's power grid in 2016, one that appeared designed to not merely cause a blackout, but to inflict physical damage on electric equipment. And when one cybersecurity researcher named Mike Assante dug into the details of that attack, he recognized a grid-hacking idea invented not by Russian hackers, but by the United State government, and tested a decade earlier...

[S]creens showed live footage from several angles of a massive diesel generator. The machine was the size of a school bus, a mint green, gargantuan mass of steel weighing 27 tons, about as much as an M3 Bradley tank. It sat a mile away from its audience in an electrical substation, producing enough electricity to power a hospital or a navy ship and emitting a steady roar. Waves of heat coming off its surface rippled the horizon in the video feed's image. Assante and his fellow Idaho National Laboratory researchers had bought the generator for $300,000 from an oil field in Alaska. They'd shipped it thousands of miles to the Idaho test site, an 890-square-mile piece of land where the national lab maintained a sizable power grid for testing purposes, complete with 61 miles of transmission lines and seven electrical substations. Now, if Assante had done his job properly, they were going to destroy it. And the assembled researchers planned to kill that very expensive and resilient piece of machinery not with any physical tool or weapon but with about 140 kilobytes of data, a file smaller than the average cat GIF shared today on Twitter....

Protective relays are designed to function as a safety mechanism to guard against dangerous physical conditions in electric systems. If lines overheat or a generator goes out of sync, it's those protective relays that detect the anomaly and open a circuit breaker, disconnecting the trouble spot, saving precious hardware, even preventing fires... But what if that protective relay could be paralyzed — or worse, corrupted so that it became the vehicle for an attacker's payload...?

Black chunks began to fly out of an access panel on the generator, which the researchers had left open to watch its internals. Inside, the black rubber grommet that linked the two halves of the generator's shaft was tearing itself apart. A few seconds later, the machine shook again as the protective relay code repeated its sabotage cycle, disconnecting the machine and reconnecting it out of sync. This time a cloud of gray smoke began to spill out of the generator, perhaps the result of the rubber debris burning inside it... The engineers had just proven without a doubt that hackers who attacked an electric utility could go beyond a temporary disruption of the victim's operations: They could damage its most critical equipment beyond repair...

Assante also remembers feeling something weightier in the moments after the Aurora experiment. It was a sense that, like Robert Oppenheimer watching the first atomic bomb test at another U.S. national lab six decades earlier, he was witnessing the birth of something historic and immensely powerful.

"I had a very real pit in my stomach," Assante says. "It was like a glimpse of the future."

This discussion has been archived. No new comments can be posted.

'How 30 Lines of Code Blew Up a 27-Ton Generator'

Comments Filter:
  • by magarity ( 164372 ) on Sunday October 25, 2020 @02:38PM (#60647320)

    How often do such devices really need connectivity for firmware updates? Really, this is a lesson in isolation and access. Admittedly, shortsighted customers will probably ignore the lesson in favor of lower costs and convenience.

    • Admittedly, shortsighted customers will probably ignore the lesson in favor of lower costs and convenience.

      I'm not disagreeing with you, however, manufacturers will release hardware with buggy software making this necessary. Or in some cases companies make it necessary for hardware to be able to phone home in order to stay functional.

      • by PPH ( 736903 ) on Sunday October 25, 2020 @02:58PM (#60647394)

        companies make it necessary for hardware to be able to phone home in order to check software license status.

        FTFY.

        • by encad ( 4448511 )

          Newest fluff, at least in Building Automation, is to also have them connected to the "cloud" to siphon off the data centrally... And most of those networks are still tightly controlled vlans because they communicate openly without encryption.

        • Reality at the hands of the bean counters reminds me of a dystopian scifi triller. Requiring online license verification for safety equipment, medical appliances and critical infrastructure like the power grid is like replacing the oceans with gasolone. How long until someone throws a lit match into the ocean of gas just to be to one who did?
        • by AmiMoJo ( 196126 )

          The real problem here is that nobody wants to accept the cost of having an isolated system.

          The manufacturer wants remote access and easy firmware update capability to keep their costs down. If it goes wrong, blame the owner for not securing their network.

          The owner wants the manufacturer to have remote access and easy firmware update capability so they can resolve problems quickly and don't have to train someone to do those things locally. They also don't want to pay more than the bare minimum for network se

        • Look, snide comment about crappy consumerism aside, no, industrial equipment does not phone home. Hell many manufacturers simply tie the software license to the hardware itself. And you buy a software license whether you want one or not.

          • by PPH ( 736903 )

            industrial equipment does not phone home

            I imagine that most of it does by now.

            And you buy a software license whether you want one or not.

            Periodically. And if you forget to renew it, it quits working when the license expiration date rolls around. And the more we rely on software as a service, it phones home every time you log in, just to make sure your account is still paid up.

      • The whole "demo" was pretty stupid, disable safety features and then show you can run a generator to destruction. You can get the same effect on almost anything with heavy moving parts by tossing a handful of gravel into the appropriate opening. Great demonstration of the cyberwarz in order to get funding for whatever it is you want funding for, but not much else.
    • Such devices should be connected for monitoring but not for control (i.e. read-only).

    • because you don't have to pay somebody to go around checking & maintaining them. A few decades ago it wasn't enough money to be worth it, but there's been so much consolidation in the industry that squeezing a little bit of money out scales in ways it didn't.

      Think of it this way, if you have 10 employees and you save 1% on labor costs it's probably not worth the risk. If you have 100,000 it starts to look mighty attractive.

      There's all sorts of crap like this. A buddy of mine drives a school bus
      • "Basically we're all being squeezed by efficiency experts..." Guess it's time for some security experts to squeeze the efficiency experts, eh?

    • There's only so much convenience you can have. A smart grid requires connectivity. It's one thing to berate a company with a co-located maintenance and control room for not isolating something, it's quite another for widely disperse infrastructure which effectively necessitates connectivity in the modern world.

      I mean you could take this to a logical extreme, the air separation station which used to supply us with pure oxygen was not run locally. There was only one field operator there to hit a red button ev

    • If you have uncleared people poking around your generator controller, you have bigger problems. Firmware also needs to be signed by the creator and verified by the device's controller before being loaded. Seems like grade school security issues.
    • Connectivity is usually provided for support, the companies that buy industrial equipment want to buy it as a service that includes the maintenance and often operation. The manufacturer either needs remote access, periodic visits or someone permanently on-site. The latter two are far more expensive and inflexible options.

      Ideally the kit would be reporting only, but often that means "our intention is that it does reporting only" and a lot of the time that's just decided after it was developed.

      One of my old c

    • Protection relays aren't networked for firmware upgrades and anyone connecting them to the internet is insane (not saying it doesn't happen). They need to communicate to the supervisory and control systems so that the plant/grid can be properly managed. In theory these networks should be isolated but in practice this is very hard to achieve.
  • by Joce640k ( 829181 ) on Sunday October 25, 2020 @02:44PM (#60647330) Homepage

    Is "GIF" the new unit of file size? What happened to "Libraries of Congress"?

    • Right. It's just like saying "it was no longer than a piece of string".

    • by DontBeAMoran ( 4843879 ) on Sunday October 25, 2020 @02:53PM (#60647372)

      Not only that, but you can have hand-drawn GIF files under 1KB and converted-from-video GIF files above 10MB.

      I'm not sure why people keep making over-simplified comparisons as if all readers are pre-school children. Let them learn what "140 kilobytes" mean - after all it should be part of basic knowledge in today's society, just like kilograms and kilometres.

      • Let them learn what "140 kilobytes" mean - after all it should be part of basic knowledge in today's society

        Yeah, but even if everyone knows it's 143360 bytes, some marketing drone at a shady disk maker invents a novel way to cheat and says it's only 140000. Which is not that novel -- like, US 2x4 lumber is really 1.5x3.5, etc.

        • by DontBeAMoran ( 4843879 ) on Sunday October 25, 2020 @03:07PM (#60647418)

          Remember that 14KiB != 14KB.

          K has always meant 1000, it doesn't matter if the computer industry twisted it to mean 1024 or that your operating system reported the wrong values for four or five decades, K has always equaled 1000 this whole time.

        • by DontBeAMoran ( 4843879 ) on Sunday October 25, 2020 @03:14PM (#60647436)

          The International System of Units (SI) [wikipedia.org] defines the prefix kilo as 1000 (10 exp3); per this definition, one kilobyte is 1000 bytes. The internationally recommended unit symbol for the kilobyte is kB.

          In some areas of information technology, particularly in reference to digital memory capacity, kilobyte instead denotes 1024 (2 exp10) bytes. This arises from the prevalence of powers of two in memory circuit design.

          So basically, Your username literally means "1000 bytes". If you wanted it to mean "1024 bytes" it should have been "Kibibyte".

          • by KiloByte ( 825081 ) on Sunday October 25, 2020 @04:08PM (#60647612)

            So basically, Your username literally means "1000 bytes". If you wanted it to mean "1024 bytes" it should have been "Kibibyte".

            My username predates that "1000 bytes" lawsuit and efforts to redefine the unit.

          • by djinn6 ( 1868030 )

            That's retarded. The SI unit of information should be the bit, not byte. Same stupidity as using kilograms as the base unit.

          • by AK Marc ( 707885 )
            there are multiple ounces. There are multiple gallons.

            There are multiple kilos.

            Your mental inflexibility isn't proof someone else is wrong. kilobytes is 1024 bytes. It isn't kK, which would be 1000. It's kB, which isn't a metric measurement, so, obviously, doesn't conform to metric standards.
        • And 2x4s are two inches by four inches (in the US). Not.

      • At least they said "gif" and not "jif".
        • Yeah, that's another debate entirely. I'm on the GIF side, though. GIF stands for Graphics Interchange Format and I don't care what the creator of the format says. Acronyms are not words, they don't follow syntactic or linguistic rules. "It's GIF but it's pronounced JIF" is just plain idiotic bullshit.

        • Actually you know what? Someone at Apple, Google or Microsoft should create a .JIF format just to block the "GIF is pronounced JIF" idiocy, once and for all.

    • by Entrope ( 68843 )

      They found that mentioning that the 30 lines of code averaged almost 5 kilobytes per line made real software people roll their eyes in disbelief. So they bury the fact that they needed 140 kB of code to do this, and headline fitting that into 30 lines.

    • by Kekke ( 236130 )

      They where all zipped :(

  • Breaking machines by using them in ways they were not designed for. Thatâ(TM)s a new one... An Oppenheimer moment rightfully so.

  • by PPH ( 736903 ) on Sunday October 25, 2020 @02:49PM (#60647348)

    It is theorized that operators watching porn resulted in a pipeline rupture and fire [wikipedia.org] which killed three people. Nothing was ever proven as the operators (having admin privileges) wiped the logs and pled the 5th Amendment during the investigation.

    Why these things need to be connected to the Internet I'll never understand.

    • by Jeremi ( 14640 ) on Sunday October 25, 2020 @03:55PM (#60647600) Homepage

      Why these things need to be connected to the Internet I'll never understand.

      You're right, they don't need to be; you can always instead physically send somebody out to (middle-of-nowhere, Alaska) with a laptop to diagnose/adjust/reprogram them in person, whenever something goes wrong.

      As to why they often are connected to the Internet, it's because nobody wants to be sent out to (middle-of-nowhere, Alaska) with a laptop to diagnose them in person -- particularly if the equipment in question is balky and has be re-serviced several times a week.

      • it's because nobody wants to be sent out to (middle-of-nowhere, Alaska) with a laptop to diagnose them in person

        It's worse than that. Nobody wants to wait at home with the lights off until someone at some company sends a tech out to the middle of nowhere either. This connectivity is as much driven by customer and regulatory demands as it is cost savings.

      • by PPH ( 736903 )

        nobody wants to be sent out to (middle-of-nowhere, Alaska) with a laptop

        If your only tool is a laptop ....

        Having worked in the utility biz: odds are very good that what needs fixing is hardware. So you head out with a toolbox, spare fuses and other consumables. And a laptop. Even if the software can tell you which subsystem is in trouble, it can't tell you which part or parts have failed. You go up with some test equipment and chase down the fault in the circuit.

      • There are efficiency advantages too. If you're getting a data feed from the devices constantly then you can do condition monitoring and fix things pro-actively when they need doing, rather than based on manufacturers service intervals or just when it breaks, it's big business these days.

        Also works the other way as can reduce maintenance by only doing it when necessary, for things like power stations that have huge outage windows then this can make a big difference.

      • That's a false dichotomy. "able to connect remotely" is not the same as "reachable over the public internet".

  • There is no such thing as a an M3 Bradley tank. There is a M3 Bradley Cavalry Fighting Vehicle. It is also not a tank; the point I care more about. A tank is something the infantry in the M3 Bradley will go over to at 4am and ask if the tank can be started so they can get warmed up.

  • by Cyanosis ( 7381552 ) on Sunday October 25, 2020 @02:57PM (#60647390)
    Alright so here is the reality of the situation. Yes the magnetic pickup signal was most likely altered for the governor on the engine unit with the code. THE MECHANICAL PROTECTION THAT HE SPOKE OF AND REFERRED TO AS THE RUBBER GROMMET WAS LITERALLY FUCKING DESIGNED TO DO EXACTLY WHAT IT DID. Its called a love joint, and in every single generator produced in the united states above 15KWH has one of these. Personally replaced hundreds of them in multiple generator sets. Made the account just to discredit this absolute idiocy. Effective? Sure, on the short term. But a small team of technicians can repair and replace this in less than a few hours with the proper equipment.
    • by gweihir ( 88907 )

      Sensationalist bullshit is far easier to produce than something actually accurate. And, you know, when you try to find out how things actually work, you may find that the engineers doing the design for these are not completely incompetent and have some non-software safety mechanisms in there...

    • by nagora ( 177841 )

      Effective? Sure, on the short term. But a small team of technicians can repair and replace this in less than a few hours with the proper equipment.

      Right. So if you organise a mass attack on 3000 of these things, how many teams are there to go around?

      • Just about every single plant has a team of on call technicians in the united states. Consider how quickly our infrastructure gets repaired after storms. Its NOT that big of a deal. Electricians Quick Facts: Electricians On-the-job Training Apprenticeship Number of Jobs, 2019 739,200 Job Outlook, 2019-29 8% (Much faster than average) Employment Change, 2019-29 62,200 Just saying. Thats just last years projections on the staffing of junior tech's.
    • by DRJlaw ( 946416 )

      Yes the magnetic pickup signal was most likely altered for the governor on the engine unit with the code.

      But you can't prove that, can you? You may replace grommets, but I bet that you can't tell me that the representative governor settings are enough to prevent the love joint from failing or shaft and/or generator damage after the love joint is destroyed. So it takes 60 resets instead of 3. Given the cycle time here, that's not a lot of time to react.

      Effective? Sure, on the short term. But a small team

    • What the article described was the generator being burned out, even without the mechanical damage it would have been scrap. In my lessons at university, I learned that a generator like this has to be in sync with the network before connecting. That means that the voltage from the generator has to match the voltage in the grid in frequency and phase.
      If it is connected when not in sync, extremely large currents will flow through the generator, and those will create huge forces as well. Large currents heat thi

    • Its called a love joint

      Lovejoy? As in the brand name that has become one of the standard coupling designs?

    • Made the account just to discredit this absolute idiocy.

      What did you discredit? The power went out. A team of technicians will take more than "a few hours" just to reach most generators in service. I see your comment about "every single plant has an on call team". LOL mate you worked in a very sheltered and very lucky industry if you think that is the case. Now back in the wider industrial reality there are whole industries where said electricians could be whole charter flight away.

  • not a problem with the generator. The subnet should have been isolated and encrypted. There is nothing wrong with remote management from designated origins with verified credentials.
    Bad Design, Bad Security Implementation and Shoddy IT Work overall. I wonder if this was in the real world.
    • by Luckyo ( 1726890 )

      All of this is done in most cases, though you'll obviously find someone who cut some wrong corners.

      But in most cases, payload delivery is not "over the internet" but by a subverted local agent inserting media that will deliver the payload into the network directly. Subversion is done by methods of the kind Snowden described from his time with CIA: arrange an accident where target person is the guilty party, have a recruiter "save" target person from responsibility to create deep sense of debt. Proceed from

    • The subnet should have been isolated and encrypted.

      Yep. Firewalls and network gear are impenetrable. There's never been many reported cases of isolated networks being reconfigured and traversed by attackers over the years, no sirree.

      Defense in depth my friend. Just isolating and being done with it is sloppy. Even the best isolations can be broken (yes, including physical ones).

  • 1. being remotely update-able. Critical systems should have a physical electric switch to enable writes to the ROMs. 2. having a fail safe system that depends on software, and software connected to the internet
  • This is like having a good old fashion beheading to make sure a knife can take off someone's head. You know, because we need to be 100% certain. The coders and engineers at the company who built the generator know that without proper limits and electronic controls, the thing will definitely burn up/blow up. If asked nicely, I'm sure they could have told the US government, the thing would fail.

    That fact that large industrial equipment will fail when run improperly is hardly a fact that needs proving. Thi

  • by nospam007 ( 722110 ) * on Sunday October 25, 2020 @03:26PM (#60647490)

    640Kbyte IS enough for everyone!

  • I don't understand, the safety relays of an industrial machine would be connected and writable? Are there really designs like that int the wild? I understand monitoring being connected, but surely safety relays would need something like a ROM chip replacement to ever get updated. Otherwise it was just a horrible design.
    As for the "Oppenheimer" moment? I can't even respond to so much cringe...

  • by tiqui ( 1024021 ) on Sunday October 25, 2020 @04:49PM (#60647720)

    It's our duty, as people [most of us, anyway] who understand how computers and networks actually work, to tell the people around us who do not, the following things:

    1. Stories like this are sensational garbage; they're aimed at those ignorant in the tech, and designed to make sales, attract clicks, and/or manipulate public opinion to pave the way for some regulation/legislation that SOMEBODY wants.

    2. Anybody who hooks critical infrastructure to the internet is a MORON.

    3. Anybody who designs a critical system into which ANY foreign code can get loaded is a MORON.

    4. The Movie "Independence Day" is total fantasy - there's no such thing as a virus that can be run on all architectures (including unknown architectures) and certainly no way for such an imaginary-virus to narrowly target a piece of unknown mystery hardware and control it to the point of its own destruction.

    Let's face it: we all know too many programmers who, having been educated to see computers as abstract black boxes, cannot write code to properly control a bit of hardware, even when they have access to all the documentation for a system, including schematics and component data sheets. The idea that a properly designed bit of hardware is vulnerable to destruction at the hands of code written by somebody without access to, and without detailed documentation for, that hardware is pure bunk. The imaginary killer virus would need to be written to execute on the particular processor architecture, know which bits to manipulate in which registers/ports/peripherals to manipulate, which ways to manipulate them, including any timing, masking, inversion, enabling etc, and in some cases which peripherals or buses to go through to get at the ports/registers that need to be manipulated, and so on. There's simply no way to do this without detailed knowledge of the targeted system. Furthermore, any safety-critical system is constantly checking its own code and will shutdown safely if anything attempts to corrupt that code - good luck getting a virus in there in the first place. Incidentally, this is all a good reason to not use COTS stuff in critical infrastructure...

    Every time somebody claims to have proven this stuff (without being a total fraud) there is always a disclaimer or some other fine print that should be examined, in which there is an admission of the ways things had to be rigged or manipulated to make the so-called demonstration "work".

    • You don't understand what they did. They disconnected the generator from the grid, waited a short while until it had lost synchronization and reconnected it. Connecting a generator when it's too far out of sync destroys it. That is not some arcane internal detail that they manipulated. They flipped a switch off and on again. There are standards for controlling industrial equipment and lots of that stuff is self-describing. Need I remind you that someone (most likely American spooks) destroyed centrifuges in
    • 1. Stories like this are sensational garbage; they're aimed at those ignorant in the tech, and designed to make sales, attract clicks, and/or manipulate public opinion to pave the way for some regulation/legislation that SOMEBODY wants.

      2. Anybody who hooks critical infrastructure to the internet is ...

      Okay you had me at number 1. The story is sensational garbage so I'll just connect my critical stuff to the internet. Thanks for the consult.

      I really hope it's not your job to explain stuff like this to other people. You start by dismissing the problem. Then proceed with dismissing end user requirements while calling them a moron. Then you made assumptions about the nature and origin or protection of code. And proceeded to downplay the idea of a virus attacking multiple machines, despite that being exactly

  • ...in 2016" Would would be the second attack on Ukraine's power grid after the Soviet Union forced their dangerous reactor design on Chernobyl and other sites.

  • by dohzer ( 867770 )

    Why don't power-stations simply use a VPN connection? They offer total internet privacy and security using military-grade encription!

  • It was out in November 2019.
  • Is this a novella?
  • Reminds me of Stuxnet as covered in documentary film Zero Days. Recommend.
    https://www.imdb.com/title/tt5446858/
  • Assante also remembers feeling something weightier in the moments after the Aurora experiment. It was a sense that, like Robert Oppenheimer watching the first atomic bomb test at another U.S. national lab six decades earlier, he was witnessing the birth of something historic and immensely powerful.

    Or have the safety systems standalone and disconnected from the network. Problem solved. If only nukes were so easy to stop.

  • Comment removed based on user account deletion
  • In electronics and electrical engineering, a fuse is an electrical safety device that operates to provide overcurrent protection of an electrical circuit.

    It is a sacrificial device; once a fuse has operated it is an open circuit, it must be replaced or rewired, depending on type.

    I have seen equipment fail and not the fuses. It's rare. Technicians jokingly might blame it on Murphy's Law. The circuit will fail to protect the fuse.

    I believe that a few good engineers out there try to design to prevent a

  • Friends don't let friends use Windows. Not on critical equipment or anything you care about.

The trouble with being punctual is that nobody's there to appreciate it. -- Franklin P. Jones

Working...