Slashdot is powered by your submissions, so send in your scoop

 



Forgot your password?
typodupeerror
×
Security Power

Data Storm Caused Nuclear Plant To Shut Down 178

rs232 writes to let us know that the US House of Representatives Committee on Homeland Security called this week for the Nuclear Regulatory Commission to further investigate the cause of excessive network traffic that shut down an Alabama nuclear plant. Investigators want to know whether the data storm could have been initiated from outside the plant.
This discussion has been archived. No new comments can be posted.

Data Storm Caused Nuclear Plant To Shut Down

Comments Filter:
  • by Clockworkalien ( 1099495 ) on Saturday May 19, 2007 @05:02PM (#19193269) Homepage
    All of the plant employees were looking up Starcraft 2 news.
    • by neoform ( 551705 )
      musta been the slashdot effect..
    • Storm in the tubes (Score:5, Interesting)

      by cyberianpan ( 975767 ) on Saturday May 19, 2007 @06:36PM (#19193959)
      I've worked in IT a while now & have never heard of a "data storm". This reminds me of

      And again, the Internet is not something you just dump something on. It's not a big truck. It's a series of tubes. Ted Stevens [wikipedia.org]
      We have plant managers concocting an odd metaphor that will only further confuse senators. Why can't they just use actual language - is it because they are deliberately trying to confuse the issue to avoid blame ? The same way the red herring of terrorism is being floated re this ? In fact it is more serious that

      1) They can't describe what happened

      2) They can't tell if outside interference, whatever the nature occurred

      3) That this might have an internal/design cause
      ... than if "terrorists" did it.
      • by ichigo 2.0 ( 900288 ) on Saturday May 19, 2007 @07:49PM (#19194355)
        Because "spike in network traffic" sounds lame. Data storm, OTOH, sounds cool and dangerous. Contact Jack Bauer quickly! We need to open a new port for the nucular plant, so the terrorists don't destroy us! And while you're at it, give us more money so we can prevent these awful storms in the future!
      • Re: (Score:3, Insightful)

        by Jugalator ( 259273 )
        I've worked in IT a while now & have never heard of a "data storm".

        Maybe it's the precursor to a logic bomb [wikipedia.org]!

        Wow, can't you request article deletion from Wikipedia on the basis of "ridiculous term"?
        Or better yet, mind erasing for the very same reason... :-p
      • Re: (Score:2, Informative)

        by binarysins ( 926875 )
        I usually hear them called packet storms, but they happen and "storm" is usually somewhere in the description. In fact, we were just troubleshooting exactly that at my work last week and the network admin used the exact phrase "packet storm".
      • 4) Apparently the computers which control a nuclear plant are connected to the public Internet, allowing anyone in the world to send them commands, viruses, or random garbage, therefore allowing them to gain remote control over the reactors. Oh, and according to TFA, another nuclear plant runs Windows (since it was hit by the Slammer worm).

        Someone please tell me that I'm wrong and the people who design these plants aren't this stupid. Please ?

        • by Splab ( 574204 )
          I don't see the big deal. Putting the controller on the internet makes sense, just like you put ATMs and bank transactions on it. It's not like theres a big red button when you connect to it that says "Commence meltdown". Nuclear reactors might have software to set some of the controls, but they definitely have hardware fall back that can overrule the software decision.

          One should never ever count on software to handle emergency shutdown of anything - you simply cannot risk having the emergency shutdown wait
        • Re: (Score:3, Informative)

          by bloobloo ( 957543 )
          The plant I'm working on the design of at the moment will have a VPN connection so that we can monitor it's performance from abroad. Running private cables over 7000 miles would not be feasible.
        • Re: (Score:3, Informative)

          by RockDoctor ( 15477 )

          4) Apparently the computers which control a nuclear plant are connected to the public Internet, allowing anyone in the world to send them commands, viruses, or random garbage,

          Might I recommend you to RTFA?
          The "data storm" appears to have been on a internal network (not seemingly connected to anything apart from other internal networks), where a data acquisition and control device barfed on some bad data and started to spew garbage onto the network. Inadequate data validation combined with inappropriate or i

      • by Anon99 ( 1103597 ) on Sunday May 20, 2007 @02:41AM (#19196301)
        >I've worked in IT a while now & have never heard of a "data storm".

        I used to work as embedded developer, and we used that term.

        It was used in embedded communications when one or several devices went bonkers and flooded common bus.
        Bit like packet storm, but without IP or other packet protocol, so it was called data storm.

        It stands to reason that in nuclear plant there are a lot of old fogeys, so company jargon might be bit outdated and odd sounding to outsider.
        • by PPH ( 736903 )

          It was used in embedded communications when one or several devices went bonkers and flooded common bus. Bit like packet storm, but without IP or other packet protocol, so it was called data storm.

          Back in the old days, this happened (occasionally) with old coax-based ethernet LANs. Decent 10Base-T hubs have provisions for blocking individual devices that 'go nuts' and keep them from screwing up the entire network. Its still possible for a single device to lock up, but it would be a very poorly designed net

      • is it because they are deliberately trying to confuse the issue to avoid blame ?

        That is the truth. 25 years ago when deploying ArcNet and later thin net we isolated our production systems, while we didn't have in-securable OS issues back then we realized PC/user behavior was not worth the risk to automated machinery. Today, it is more critical than ever.

        But the gross incompetence here goes to management. They should all be fired with cause and with prejudice and shut the system down until it is design

      • If you've worked in IT for a while, you would remember Procomm. It was probably the best PC based terminal emulation/RS-232 scripting programs. I know a few folks who still use it for automated embedded equipment telemetry & command applications. Procomm was written by none other than "Datastorm Technologies corp.".

        Now back on topic: If you RTFA it says that there was an embedded networked controller that was "babbling" (flooding the internal network with unwanted traffic). Unless some hacker from o
      • by vakuona ( 788200 )
        Because data sounds technical, and storm sounds bad.

        Two key ingredients if you want people to
        a) Think you know what you are doing and,
        b) Scare them enough into doing something.
  • Shut down? (Score:5, Insightful)

    by Anonymous Coward on Saturday May 19, 2007 @05:04PM (#19193281)
    >Investigators want to know whether the data storm could have been initiated from outside the plant.

    Do invesigators also want to know how a "data storm" could have caused a nuclear plant to shut down?
    • Re: (Score:2, Informative)

      by Detritus ( 11846 )
      RTFA, bozo.
    • Re: (Score:3, Informative)

      by Sj0 ( 472011 )
      It looks like it was a modbus plus network. We're talking a proprietary physical layer on up, specifically designed for PLCs to communicate with one another.

      If there was a communications problem and a PLC blinks out of existence on a mission critical system, it's only the safe thing to fail the entire system to prevent damage to people, the environment, and equipment.
    • Do invesigators also want to know how a "data storm" could have caused a nuclear plant to shut down?

      Not really, it is BS grandstanding for politics like they are doing something when they are not.

      For if it were not, the plant would have been shut down by now. They are talking this happened in August, and I would bet the problem still exists.

  • The employees used YouTube and MySpace [chron.com]
  • by SuperBanana ( 662181 ) on Saturday May 19, 2007 @05:10PM (#19193313)

    Some choice quotes, emphasis added:

    An investigation into the failure found that the controllers for the pumps locked up following a spike in data traffic -- referred to as a "data storm" in the NRC notice -- on the power plant's internal control system network. The deluge of data was apparently caused by a separate malfunctioning control device, known as a programmable logic controller (PLC).

    "Conversations between the Homeland Security Committee staff and the NRC representatives suggest that it is possible that this incident could have come from outside the plant," Committee Chairman Bennie G. Thompson (D-Miss.) and Subcommittee Chairman James R. Langevin (D-RI) stated in the letter. "Unless and until the cause of the excessive network load can be explained, there is no way for either the licensee (power company) or the NRC to know that this was not an external distributed denial-of-service attack."

    Wow. Just...wow. As if you needed more proof that this wasn't a hacking attempt:

    "The integrated control system (ICS) network is not connected to the network outside the plant, but it is connected to a very large number of controllers and devices in the plant," Johnson said. "You can end up with a lot of information, and it appears to be more than it could handle."

    Seriously, how stupid do you have to be to think "OMG, Haxxors?" Answer: work at Homeland inSecurity, or be a Congresscritter. They already figured it out. It was a controller for a specific piece of equipment that flooded the network and triggered a bug in the variable-frequency-drive controllers for pumps.

    • You missed one.... (Score:4, Interesting)

      by iknownuttin ( 1099999 ) on Saturday May 19, 2007 @05:13PM (#19193339)
      FTFA: "What is happening in this marketplace is that vendors will build their own (network) stacks to make it cheaper," Peterson said. "And it works, but when (the device) gets anything that it didn't expect, it will gag."

      Sounds to me that the vendors under-engineered their network and still charged mega-bucks for it. The auditors, I'm sure, are making the most out of this to justify their fee.

      Nothing to see, move along - I'll say!

    • Re: (Score:3, Funny)

      by MECC ( 8478 ) *
      Never hire windows admins brandishing the moniker "network admin".

    • by A Bugg ( 115871 ) on Saturday May 19, 2007 @05:45PM (#19193577)
      I work at a nuke plant as a system engineer. One of my systems are the reactor recirculation pumps, these type of pumps. I know for a fact there is no way hackers could "data storm" my pumps and there is extreme doubt in my mind that the same thing could happen at Browns Ferry. The pumps digital control system isn't even near any outside network.

      However, I will fully put the blame on the PLCs. Those little suckers come in handy but if you don't completely understand every line of code and every instruction they can f_ck you over.

      I also love how they say "well if you can't prove it wasn't, then it must have been".
      • by Anonymous Coward on Saturday May 19, 2007 @06:17PM (#19193821)
        You just have to love Browns Ferry don't you? This is the same plant that had wired its control cabling for two nuclear reactors through the same area. Then they had workers check the air tightness by using candles near their flammable insulation. It wasn't air tight and the flame of a candle was sucked into the insulation. Thus a fire broke out, $100 million of damage occurred, and control was lost of their two nuclear reactors for something around 8 or more hours. Why 8 hours? Because their fire team tried to fight the fire with portable CO2 extinguishers. Yes, for 8 hours. Until the local fire department (which they previously obstructed) put it out with water in 5 minutes. Idiot designers and idiot employees. I'm surprised that plant didn't have a meltdown before TMI. But boiling water reactors are a little harder to destroy.
        • ...Idiot designers and idiot employees. ...

          Driven to it by idiot management and idiot politicians.

          Going for idiot "employees" and "designers" is going after the effect, not the cause. You want idiots, you hire them. You want "cheap" designs, the designer will design them. Employees and designers are told what to do. If you push back in these roles too much you will not have a job. Mind you, I would include the primary contractor.

          Fixing this means fixing management. The best way to do this is to sh

      • by Bo'Bob'O ( 95398 )
        What I'm wondering, is are these even on TCP? Or for that matter even Ethernet? I would have thought this would be something more like RS485 to begin with.

        Though I do notice that it says the plant has been closed for a decade until this year. I guess it's best to check all options, but how is that tiny possibility remotely any sort of news when compared to the vastly more likely problem of a bug in a brand new control system, especially when it seems they found exactly what that bug was already?
    • It's not stupid. (Score:5, Insightful)

      by twitter ( 104583 ) on Saturday May 19, 2007 @05:51PM (#19193625) Homepage Journal

      Seriously, how stupid do you have to be to think "OMG, Haxxors?" Answer: work at Homeland inSecurity, or be a Congresscritter. They already figured it out. It was a controller for a specific piece of equipment that flooded the network and triggered a bug in the variable-frequency-drive controllers for pumps.

      As someone who used to work in system's engineering for a sister BWR, I think the inspection is a good idea. Oh, there's dumb and there's nuclear dumb but this is not a case of either. Nuclear dumb involves putting machine guns nests inside the plant. Finding the root cause of the accident is a good idea.

      Handwaving about a PLC device won't do. What ultimately caused the PLC malfunction needs to be answered at a component level. There's going to be something wrong with it and that should be reported and every other device like it needs to be ripped out and trashed. If there is not component failure, there's a software problem which also must be understood.

      Yes, it could have been hackers. The "internal control network" might at some point hits a desk that's connected to the wider world. It could be something mundane and unintentional, like an operator's virused up laptop.

      An outage like that is something that's going to have both NRC and corporate ass-chewers looking at everything. Corporate might want to paint a nice picture for the NRC, but the poor devil that lies to them goes to jail. In either case, the problem will be identified and eliminated.

      You might also have noted in the article that this is not the first plant to go thumbs down over some winblows born virus. In 2003, the slammer worm caused havoc at an offline Ohio plant [securityfocus.com]. Yes, that was hackers. They did not mean to do it, but the plant's systems were open to it and failed. That's not acceptable from any standpoint.

      Despite the better advice of the computer people at the plants, Entergy is a big M$ Partner. They take the big dogs out fishing and sell them the works. Ten years ago, M$ had something worth while and interesting. It was used in places it should not have been. Worse, the flaws from ten years ago have not been addressed or fixed. A good clean up is in order.

      • Wow, someone who's worked in a BWR down modded in less than ten minutes. Nice work, trolls.

      • insightful. He's worked in a nuclear power station and seems to be clueful.
      • by Z00L00K ( 682162 )
        In general, software installations at industrial plants has to have a lifetime of decades. This means that even if the software selected isn't the cheapest initially it may prove to be the cheapest in the long run since it will have to be stable. Each unplanned stop of a large plant due to an undefined malfunction will take time and cause costs in the million dollar class.

        One way to safeguard against data storms to some extent is to create network segmentation so that a malfunction in one part of the plan

    • by jd ( 1658 ) <imipak@ y a hoo.com> on Saturday May 19, 2007 @09:31PM (#19194909) Homepage Journal
      I believe this is the nuke plant that is supposedly using Windows NT to handle SCADA (Supervisory Control and Data Acquisition) functions, and I know it's the plant that has shown gross incompetence in relation to fires (see assorted other postings). Internal systems are not supposed to be connected to external networks. It's unlikely this one was - not because they're smart, but because I'm not certain they'd know how. We can therefore eliminate external causes. Sadly, we have passed from the Age of Englightenment and the Age of Reason into the Age of Paranoia and the Age of Dementia, so that is likely the attribution we can expect from the Department of Homeland Insecurity.

      A random fluctuation in internal traffic levels seems equally unlikely. Why? Because it has worked for some time, and I doubt the reactor was doing anything unusual at the time. A true network storm is unlikely - the term exists, but describes an astronomically rare situation. If a network is flooded, it is either near or at capacity. A network storm is when capacity is exceeded in a way that is self-perpetuating. The last time I remember the term being used in a public forum was I think over twelve years ago when a public demonstration of the multibone caused a cascading router flap that shut down a large segment of the Internet backbone due to total gridlock. It wasn't just that nothing else could get through - nothing AT ALL could get through.

      What does this leave us? It makes it extremely unlikely that the network traffic per se had anything to do with the shutdown. Much more likely is a cumulative error in the devices involved that merely happened to turn into a fatal bug at roughly the same time as the network spiked. It might be network related, but nobody here can seriously believe it was network caused. Networks may be polled, in which case network traffic that escapes being polled is simply never seen. Network drivers may also be event-driven, but if the interrupt handler is buggy - which would usually mean the handler can be interrupted by itself indefinitely - it's hardly the fault of the network.

      In other words, this is a gross programming error that the coders and managers are desperately trying to blame on something - anything - other than their own ineptness. It might merit Scott Adams making a Dilbert cartoon over, but that's it.

      • Re: (Score:3, Insightful)

        A random fluctuation in internal traffic levels seems equally unlikely.

        Look up "Poisson distribution". At low packet rates, large rate fluctuations by random chance are the rule. You also have to consider events that can trigger a common packet rate spike, such as a a non-critical subnet being power cycled. Combine this with a device that has an overflowable packet buffer and you have a recipe for inevitable failure.

        A true network storm is unlikely - the term exists, but describes an astronomical

      • Re: (Score:3, Insightful)

        by kasperd ( 592156 )

        A random fluctuation in internal traffic levels seems equally unlikely. Why? Because it has worked for some time, and I doubt the reactor was doing anything unusual at the time.

        This is not about the network being highly loaded with lots of packets comming from all sorts of places. This is about a single device for some reason flooding the network. I have seen the results of units flooding a network with broadcast traffic. I don't consider it highly unlikely for one unit to eventually start doing that becaus

      • Re: (Score:3, Insightful)

        by DerekLyons ( 302214 )

        A random fluctuation in internal traffic levels seems equally unlikely. Why? Because it has worked for some time, and I doubt the reactor was doing anything unusual at the time. A true network storm is unlikely - the term exists, but describes an astronomically rare situation.

        When investigating an accident you cannot ground rule out an occurence that is unlikely or rare - unless you have positive evidence that said unlikely or rare condition did not occur, or positive evidence of another cause. "Unlikely"

    • What I want to know is what the hell are Homeland Security doing trying to protect Nuclear Power plants when there are still 11 yr old girls living in housing projects downloading Britney Spears music off the net FOR FREE!! They should obviously be focussing on funnelling money into the pockets of the people with whom Britney has a contract with.
  • Standards! (Score:5, Insightful)

    by 26199 ( 577806 ) * on Saturday May 19, 2007 @05:12PM (#19193331) Homepage

    You'd hope that in something as critical as a nuclear power plant the answer would be, very quickly, "no, it didn't come from an external source because that's impossible". Followed by detailed analysis of the logs to determine which internal system screwed up.

    That said, the article is a bit sparse on actual technical details, so my derision may be unwarranted.

    • Re: (Score:3, Interesting)

      As it should be. The point is this. Any of the computer/network equipment that actually runs the plant should not be connected to the outside, period. All normal computers for office work, typing up non-classified reports and reading slashdot should be on a whole seperate network. Idealy, there should be 3 networks since the plant should only have certified equipment connected to it that won't cause what happened here to take place unless something was truly malfunctioning. I'd be a little scared to fin
    • Re:Standards! (Score:5, Insightful)

      by mrchaotica ( 681592 ) * on Saturday May 19, 2007 @05:37PM (#19193507)

      You'd hope that in something as critical as a nuclear power plant the answer would be, very quickly, "no, it didn't come from an external source because that's impossible".

      Actually, power plants have to have a connection to the outside world. Why? Load-balancing for the power grid. If another plant goes down somewhere, this plant needs to know about it so that it can adjust output to compensate. For that, all the plants need to be hooked to a communications grid, which could conceivably be hacked (even though -- I would hope -- it's not connected to the Internet).

      • Re:Standards! (Score:5, Interesting)

        by Artifakt ( 700173 ) on Saturday May 19, 2007 @06:47PM (#19194015)
        This actually can be avoided (and AFAIK current designs do). Fast, electronic level response to avoid blackouts and such requires very much less time than changing reactor output would either allow or facilitate anyway, so the direct machine to machine communication links don't really need to go to the power cycle control systems at all. Instead, rapid response grid balancing is done at external switchpoints. For the newer designs, these are outside the whole plant at substations, let alone just outside the core areas. Between these links and reactor control systems, there's supposed to always be an air gap.
                Given that, any hacking would have to include a social engineering element designed to fool the operators into making the wrong decisions. If we include that stipulation, yes, it's quite conceivable. If we postulate someone bridging the air gap, maybe by something as simple as hooking a laptop that also contains a wireless card into the control network, then a non-social engineering attack becomes conceivable, but not really otherwise.
                DOE and NRA doctrine is that adjusting reactor output based solely on a trigger event outside the core instrumentation is supposed to always require a high level human decision. Supervisors are also at least supposed to be trained to the point where they can make these decisions without adding any more response time than a conventional, (i.e. hydroelectric or coal based), plant would need for their human level decision events. (Yes they have them. For example the four TVA dams that supply Alcoa aluminum face a whole series of individual and joint human level decisions every time Alcoa's main furnace system glitches, and these have to include how long Alcoa expects them to need to dump power elsewhere, and for each of them, what options the other three dams are considering).
              The DOE does not legally presume that reactors are even as responsible for balancing the grid as conventional plants, but given how much older a lot of the conventional plants are, it's pretty easy to do much, much better than is strictly required, and it should be noted that, in the last New York blackout all the cascade effects and switching failures happened in 1940's era or earlier fossil fuel plants, and the worst points were 1930's or even 1920's era designs. Still, the rules are that if the conventional plants are failing at load balancing, even if the grid is experiencing severe cascade failures, the nuclear sites will let the whole thing crash rather than take the risks of trying to stabilize the grid by actually modulating their reactions.
        • Re: (Score:3, Interesting)

          by dbIII ( 701233 )

          Fast, electronic level response to avoid blackouts and such requires very much less time than changing reactor output

          It's called hydro - or sometimes even pump storage. Conventional thermal power is cheap but it takes a long time to increase output unless there is already spinning reserve. Non-conventional thermal power still takes time BECAUSE IT IS NOT MAGIC unlike what we are led to believe by those that want to build a few hundred 1950's style plants painted green. Nuclear power possibly would be a

          • As one of those who would like to see hundreds of new nuke plants, I'll point out that nuclear power has actually slipped slightly below coal for production costs [nei.org]. In addition, new pollution control requirements and the possibility for CO2 sequestration requirements substantially increase the cost and slightly reduce efficiency in coal plants.

            I try to always point out that I'd use the nuclear power to replace coal power, which takes nearly as long to cycle up or down, not to try to replace more responsive
            • Re:Standards! (Score:5, Informative)

              by dbIII ( 701233 ) on Sunday May 20, 2007 @03:27AM (#19196439)

              As one of those who would like to see hundreds of new nuke plants,

              After some R&D and building some prototypes of promising new designs I'd be right with you - but our current best bets are things out of South Africa (pebble bed) and India (accelerated thorium) done on very small buidgets with very small teams and they need more work. The mainstream is just chasing taxpayer supplied pork. If they were after more than a handout they would be putting in some effort - instead they spend orders of magnitaude in PR, advertising and outright bribes than R&D.

              As for costs - you can't just conveniently ignore capital costs. If you could hydro, wind, solar etc would win every time even in those places where it would be a stupid idea or where the capital costs are far too large for the return. Nuclear power is a possiblity in those places that have the infrastucture of a weapons program but everywhere else you would have to build up an entire industry from scratch. Iran is the best example currently where that is taking place and it has cost them a fortune to do so - hence few people think it is for purely civilian purposes there. In South Africa it was possible to take people from the weapons program to develop pebble bed. It is also far too big an investment for private enterprise - hence no new plants getting built while governments had cold feet on the issue and the "new generation" designs from companies like Westinghouse are just tweaked 1950s designs painted green.

              • As for costs - you can't just conveniently ignore capital costs.

                Who says that I'm ignoring [uic.com.au] them? Referenced site is for European power, and nuclear comes in cheaper at 23.7 E/MWh, vs 28.1 for coal. That's including capital costs, but not including CO2 tax, which raises coal to 44.3.

                If you could hydro, wind, solar etc would win every time even in those places where it would be a stupid idea or where the capital costs are far too large for the return.

                In the USA, hydro is considered 'overutilized' IE we can'
                • by dbIII ( 701233 )

                  Referenced site is for European power, and nuclear comes in cheaper at 23.7 E/MWh, vs 28.1 for coal. That's including capital costs,

                  Now we are getting somewhere - but to see what I mean you will have to see what the assumptions applied to get those numbers are, wonder why the British experience is vastly different in economic terms despite doing everything the same way, and wonder why private enterprise never builds these things even though they are portrayed as being the winning option. Once you've done

                  • Now we are getting somewhere - but to see what I mean you will have to see what the assumptions applied to get those numbers are, wonder why the British experience is vastly different in economic terms despite doing everything the same way, and wonder why private enterprise never builds these things even though they are portrayed as being the winning option. Once you've done that you'll understand what I mean and realise you have been conned.

                    We are seeing a number of new reactor construction projects - it's
    • Re: (Score:3, Interesting)

      by legirons ( 809082 )
      You'd hope that in something as critical as a nuclear power plant the answer would be, very quickly, "no, it didn't come from an external source because that's impossible

      Indeed.

      Unfortunately, sometimes our favorite software supplier is involved... [theregister.co.uk]
    • They ran into a potentially dangerous system error, and could not immediately determine the source. If I were running the shop over there, the very first thing I would ask is "Is there any possible way this could be initiated from the outside?" Because I don't want someone to say "Of course not," and go on their merry way. I want someone to be able to explain what happened, why it happened, and what potential circumstances could possibly cause it to happen again. When I ask if it could possibly be start
  • Firstly I would re-design that entire infrastructure and rid that power plant of incompetent IT people. Secondly I would hold those in power responsible for 1) not having failover measures in place 2) not having a stable and robust enough infrastructure in place 3) obviously not being SCADA compliant. If they can't pass IT security implement simplistic measures such as a properly designed network, it makes me wonder about the physical security aspects of it. What am I paying higher taxes for everytime the g
    • Re: (Score:3, Insightful)

      by Detritus ( 11846 )
      When you get back to the real world, let us know. You don't just wave a magic wand and completely redesign and reimplement a highly complex safety-critical system.
      • A highly complex safety-critical system? Oh you're right. It's ok if it fails. Its not like its safety-critical or anything. Besides it was only the network that caused it to collapse. You're right no need to wave a magic wand and have people collaborate on setting up something that works. We'll wait for true blue self defending/repairing networks to be available and let the network fix itself. Wow. Do you by any chance work for Cisco or someone else spouting these "Next generation self defending networks"?
        • Yes, it is okay if it fails. You set the actuators up so that if they lose signal then they fail in their safe position. Then having done the design you carry out a HAZOP to make sure you've caught the problems.
    • Re: (Score:3, Informative)

      It's not the IT people PCL are coded by EE not IT people.
    • by mrcdeckard ( 810717 ) on Saturday May 19, 2007 @05:34PM (#19193491) Homepage

      i think the fact that an unforeseen erroneous condition caused the plant to *shutdown* and not *meltdown* is a pretty good indication that it was designed quite well.

      There will always be unforeseen situations. The key is for the system to shutdown in an orderly fashion. In programming, this is accomplished through use of error traps.

      Now, the hysteria surrounding terrorism is another thing the plant engineers have to worry about.

      i just wonder if and when we get to put this hysteria behind us, and get along with our lives. unfortunately, terry gilliam's brazil is on a constant loop in my mind these days. . . .

      mr c
      • by Ajehals ( 947354 ) on Saturday May 19, 2007 @08:29PM (#19194541) Journal

        i think the fact that an unforeseen erroneous condition caused the plant to *shutdown* and not *meltdown* is a pretty good indication that it was designed quite well.
        really? you think that the loss of a power plant for a period of time due to network traffic is a sign of "quite good design"?

        This might sound unreasonable but I would never expect a power plant (which has a lot of things depending on it) to shut down unless there was a major failure of a component or some other safety risk. Network traffic on its own, or its effects shouldn't ever be the cause. In a nuclear power plant you control ALL the nodes attached to the network, the nodes attached should not be in a position where they can saturate any individual node to the point of failure, especially if that failure causes a shut down of something as critical as a power station.



        I can think of times where I have seen massive network spikes usually caused by issues with routing on fairly non-trivial networks, or loops where mistakes have been made and policies have not been followed, (lack of sleep or lack of patience), but then comparing an advertising companies internal network at 3am, or a paper factories network at midnight to a nuclear power station is taking it a little far.

        There will always be unforeseen situations. The key is for the system to shutdown in an orderly fashion. In programming, this is accomplished through use of error traps.

        That would be fair if we were talking about a software failure after some sort of unforeseen environmental issue, it would even be OK if an auto plant stopped production because of an unforeseen fault, and whilst power plants should certainly fail safe, they should be robust enough that a situation where failure is the only option is extremely difficult to achieve. whatever happened to redundancy?

        Now, the hysteria surrounding terrorism is another thing the plant engineers have to worry about.
        As for the external angle terrorism or not, I doubt it. If there is a system that can be brought down by weight of traffic, and that system is important enough that failure requires a power-plant reboot (:)) then there needs to be an air-gap. Someone up thread suggested an employee's laptop with a virus as a possible method of infection.. Who in the hell allows an unchecked laptop of any description onto their LAN? never mind a network that also contains components that run a power plant!!

        I would suggest that this is hype to 1) keep terrorism at the top of everyone's agenda, and make people feel unsafe, after all that sells papers and grabs viewers (which in turn sell advertising) 2) deflect some of the negativity that this incident would produce (I wish that I could blame terrorists for my mistakes sometimes... "no that project plan... I haven't got it, but I'm checking to see if my poor time management is caused by terrorism or simply my inability to organise my resources properly") and 3) Security risks presumably attract additional funding, sureley it would be nice to get an extra few million in the next budget.

        Honestly, this probably shows a component failure and some poor design, understandable, but unacceptable in this area. If and I say If with some considerable doubt, this turns out to be, or is reported as an external event, then whoever enabled external network access to what appear to be critical systems within a nuclear power plant on the US mainland need to be identified and punished, together with the contractors who built or maintained it, the managers or consultants that assessed and managed it and the politicians who have responsibility for public safety. But as I said, it will probably turn out to be a simple component failure and some poor design.

        • sorry, i was going on the information another poster provided: it was not external network -- that is, it didn't happen over their DSL line . . .

          a sensor evidently went haywire and started to dump a ton of data out on the internal sensor network. the way i imagine it, the metric shit ton of sensors in the plant are all networked in some fashion. one went bad, generating the analog of a DOS attack. the plant SHUT DOWN. this is good. this is better than chernobyl, eg. way better.

          sure: redundancy, disc
    • by twitter ( 104583 )

      Firstly I would re-design that entire infrastructure and rid that power plant of incompetent IT people.

      You need to find the root cause [slashdot.org]. You don't know it yet, so you don't really know what to do.

      Chances are, the cause has been written up by the four or five systems engineering people in charge of the plant. They ARE competent, but they are never given the resources they need.

      Why wasn't there any failover who knows.

      There was a failover - they overrode the broken thing. Had the operators been g

    • by grumling ( 94709 )
      Well, your plan will mean that you will replace "incompetent" IT people with "inexperienced" IT people. Better to use the FAA model of accident investigation, where liability is limited and the true cause is determined (and published). That way, engineers aren't worried about being sued for a simple mistake and the entire population benefits by learning from other's mistakes.

      This story smacks of a slow news cycle and nuclear fear mongering.
  • Political FUD (Score:4, Interesting)

    by Bellum Aeternus ( 891584 ) on Saturday May 19, 2007 @05:23PM (#19193409)
    As usual, the American government is looking to extend its control over things. "Oh noes, look what terrorists might have done. Homeland security needs more funding and less oversight to prevent this in the future." When will people learn to assume the government is lying first, then wait for them to prove themselves right later?
  • by Angostura ( 703910 ) on Saturday May 19, 2007 @05:29PM (#19193449)
    Isn't it a bit odd that they were using a non-deterministic network - something like Ethernet, by the sound of it. Back in the early 90s, I was always told that networks like Ethernet were great for office apps, but not where you wanted guaranteed times for message delivery. For that token ring, FDDI and the like were better. What is the network infrastructure of choice in a nuclear power station?
    • by mplex ( 19482 ) on Saturday May 19, 2007 @08:00PM (#19194391)
      Using Ethernet is not odd, that's literally all there is these days. Sure, there are technologies like Infiniband, but Ethernet is far and away the cheapest and most widely supported networking standard. It sounds like they were experiencing a broadcast storm from a locked up device. I can't tell you the amount of times I've seen stand-alone devices lock up on a busy network because of a bad TCP/IP stack. Often times they will flood packets, especially broadcast frames. There are protections against bad devices such as broadcast limiters and a number of features that protect and limit unauthorized or undesirable traffic.

      Ethernet isn't perfect but it's the only realistic option. Managed properly, it can be very reliable. The biggest problem I see from this article is that there is a lack of regulation and testing of the equipment that goes in to these plants. These poor TCP/IP stacks should have never gotten past the testing phase when it comes to a nuclear power plant.
      • Re: (Score:3, Insightful)

        by Bo'Bob'O ( 95398 )
        This are PLCs we're talking about, there are loads of network, protocol and connection systems, proprietary or otherwise, for all ranges of complexity.
    • by Eravnrekaree ( 467752 ) on Saturday May 19, 2007 @08:16PM (#19194479)
      I find it particularly astounding that a nuclear power plant control network would have any connectivity to an external network. The article mentions the traffic flow may have come externally. That a nuclear power control system is anywhere near the internet really is quite disturbing. The article also mentions infected Windows computers contributing to the outages in 2003. I find it interesting that computers involved in electrical grid would be connected to the internet or have such lax security, and even run Windows of all operating systems at all. It really is inexcusable for security to be so poor. Simply keeping network programs running as non priveleged users in a jail one would think would be basic, to protect against exploits and systems becoming corrupted.
  • Even stupider (Score:5, Insightful)

    by packetmon ( 977047 ) on Saturday May 19, 2007 @05:30PM (#19193463) Homepage
    After yet re-reading, I find this government even more insanely stupider than I would have hoped for... Such failures are common among PLC and supervisory control and data acquisition (SCADA) systems, because the manufacturers do not test the devices' handling of bad data, said Dale Peterson, CEO of industrial system security firm DigitalBond.

    "What is happening in this marketplace is that vendors will build their own (network) stacks to make it cheaper," Peterson said. "And it works, but when (the device) gets anything that it didn't expect, it will gag."
    So you mean to tell me pretty much there is no enforcement for manufacturers to maintain compliance on their products even if those products are going into a nuclear *ANYTHING... Which on the worst case scenario could cause catastrophe, yet we have regulatory commissions on the flow of ketchup, regulatory commissions/directions/etc., on weight loss products, lipsticks, etc. (FDA), but this place is not concerned with nuclear plants. Sinful.
    • Re: (Score:3, Informative)

      by fluffy99 ( 870997 )
      This is pretty common. Also consider that the PLCs are usually custom programmed by the end-user and bad data is usually not tested by the programmers either. Heck, there are tons of commercial network devices that behave very badly when face with too much or incorrect data. Try running a full-blown security scan on your network and see what pukes. I have to go power cycle a bunch of Intel piece-of-crap print servers every time I do a port scan. Don't even get me started on the crappy snmp implementat
  • by ewhac ( 5844 ) on Saturday May 19, 2007 @05:32PM (#19193471) Homepage Journal
    People with longer memories may recall that Brown's Ferry had a massive fire [ccnr.org] a couple decades ago that burned in the wire racks underneath the reactor control room, very nearly destroying the staff's ability to control the reactor at all. It became a cause celebre among the anti-nuclear crowd alongside Three Mile Island.

    At least their reactor failed to "off" this time...

    Schwab

    • Re: (Score:3, Informative)

      >At least their reactor failed to "off" this time...

      It didn't just "fail to off", they manually shut it down. They followed procedures and placed it in a safe condition. No need to sensationalize it.
      • Only barely. Sounds like it was very close to a Chernobyl. If their makeshift coolant pump had also failed for some reason (like many others had already), then it would have been exposed core and meltdown time.

        Even in a free market democracy, people are complacent, careless, greedy, dumb and just plain human.
    • "They were also using candles to determine whether or not the leaks had been successfully plugged .. The electrical engineer put the candle too close to the foam rubber, and it burst into flame"

      Don't try and find sources of draft using inflammable foam and a candle ..

      Don't route the backup system through the same conduits as the primary one ..

      was Brown's Ferry *AGAIN!?!??!* (Score:4, AGhaaaaaa !!!, ohh God, make me a believer )
  • We slashdotted a nuclear power plant! :-)

    Behold, the power of Slashdot!

  • a cat (Score:2, Funny)

    by eille-la ( 600064 )
    A cat fell asleep on a keyboard
  • by BillGatesLoveChild ( 1046184 ) on Saturday May 19, 2007 @06:08PM (#19193745) Journal
    > data storm

    Is that a nice way of saying they were downloading pr0n?

    > US House of Representative's Committee on Homeland Security called further investigate

    Boss: "So we don't have the backups for the first two weeks in April"
    Employee: "Yes Boss. They were obviously misplaced by terrorists"

    When Homeland Security is done, my refrigerator door was left ajar last night. I think it was terrorists too. Think I'll phone this one in.
  • by rush22 ( 772737 ) on Saturday May 19, 2007 @06:13PM (#19193781)
    "Ok, techie, give me the jist of it."
    "It seems the problem was with the NC9828A chip"
    "Oh? And what was the problem?"
    "It melted, basically. It went bonkers."
    "Ah, and then what happend?"
    "Err... it caused the shutdown."
    "But how?"
    "Well, I presume the AH-982's got deluged with data, so they shut off."
    "Ah, so it was some sort of data thing."
    "Kind of, the failing chip would start sending data in the network t--"
    "Hey, it's like a storm of data! Hah! I get it!"
    "Umm, basically."
    "Oh man. A data storm! I better tell the NRC"
    "Ok, sure."

    Later...

    "Sir, I have the cause of the shutdown, it was caused what the tech guys here would call a data storm."
    "A data storm? Wow. So your reactors got a bunch of bad datas, right?"
    "Errr.. kind of, the microchips melted."
    "Data can do that?"
    "Yeah, it's like a storm on our, uh, logic networks. I guess that can melt the microchips"
    "Uh oh. Maybe this storm came from outside the plant! One of those hacker attacks!"
    "Hmmmm, the guy said it melted, but I suppo--"
    "Oh crap I better inform Homeland Security!"
    "Ok, sure."

    Later still...

    "Yeah, we had a data storm and it melted the reactor networks."
    "How did this data storm happen?"
    "I don't think they know yet, but it messed up big time."
    "My God. Do you realize this could be Al Qaeda?!!"
    "Could realize wha--"
    "Al Qaeda! Terrorists. Internets terrorists."
    "I don't know if the reactors are hooked up to the Interne--"
    "Listen. Keep this quiet, but make sure you tell everyone you know. These reactors are not safe! No one is safe from the terror!"
    "Well, it was a data storm. Can terrorists make data storms?"
    "Yes. They caused your meltdown."
    "No, no, the microchips melted down because of the storm. A meltdow--"
    "In the terror business, there's more than one type of meltdown, you just let us handle this."
    "Ok, sure."
  • /. again.
  • They device manufacturers create their own stacks and then don't test them? Are we just asking for a nuclear accident, or does this strike anyone else as stupidity?

    At least it wasn't connected to the public internet. Can you imagine the havoc THAT would create. But I wonder why they're treating this as a criminal thing. Did someone modify a device?
  • There's an excellent summary on Risks [ncl.ac.uk].
  • Good news. (Score:4, Informative)

    by Sj0 ( 472011 ) on Saturday May 19, 2007 @11:46PM (#19195601) Journal
    Great news, guys. This is going to be a non-issue. People are freaking out because a digital device is involved, and freaking out because a nuclear power plant was involved, but I do industrial control system and DCS design for a living, and I'll tell you right now, that you simply can't access control networks from the outside. There are seperate, often redundant networks, and even then, depending on the way the plant was designed, we're talking modbus plus or something that PCs don't normally access.
    • We had a customer that wanted to know if our monitoring system was protected from viruses, because they were worried about a Windows worm getting into their plant DCS via the Modbus interface. I tried to explain how that wasn't really an issue, that it would be very difficult to transmit a worm via Modbus registers over an RS-485 connection, but I was cut off with, "we know it's a problem, how are you going to deal with it?" My own thought was, "you people are going to be a problem, how are we going to deal
      • by PPH ( 736903 )

        That same principle should be applied to power plants, nuclear or otherwise, or any industrial facility where substantial damage could result from unauthorized or malicious access. I'm always amazed when I hear otherwise. You just want to grab these people by the necks and shake them, all the while asking, "Are you goofy? What's the matter with you!"

        That works fine until its the plant manager who insists on having a real time process monitoring app. running on his laptop and then be able to visit the goa

        • Grabbing and shaking such people by the neck is a carrer limiting move.

          Yeah. But it's fun to think about. Although, in some ways it's too bad our society isn't run more along the lines of the Klingon Empire. I mean, a Klingon underling wouldn't bother grabbing a boneheaded supervisor by the neck and shaking him. He'd execute the bastard on the spot for incompetence and disloyalty to the Empire ... and take his job.

          Klingon power plants are run very securely, I understand.
  • by Torodung ( 31985 ) on Sunday May 20, 2007 @02:02AM (#19196169) Journal
    Who in God's name connects a plant's coolant regulation systems to the Internet? How could it be an outside agent when the "data storm" happened on the plant's INTERNAL network.

    The article says that explicitly. "Internal network." The DHD is worried about outside agents penetrating the plant personnel, not someone with a laptop uploading a virus like Jeff Goldblum in "Independence Day."

    If there *was* such a "data storm" attack, it would _have_ to be caused by an inside saboteur. The plant needs to focus on HUMAN security, not computer security. Either that or they need to reconsider a faulty design.

    But can we try, just try, not to write completely hysterical baloney? Hysterical baloney is a tradmark of "Homeland Security," and they might see fit to sue.

    --
    Toro
  • by Esben ( 553245 ) on Sunday May 20, 2007 @04:50AM (#19196685)
    I have actually seen such a problem myself: Controllers crashing because someone was testing the network. The problem was, ofcourse, that the CPU spent a lot of time to handle the amount of packages on the network and therefore didn't have time enough for it's real-time application. (It didn't help that the platform didn't support DMA.)

    Solution: Make the network interrupt handler threaded and prioritize it below the real-time application. Sure, that doesn't help the SCADA performance, but you have to make sure that the real-time application meets it's deadlines no matter what is going on on the network. I simply don't buy that you can secure a network stretching over more than 1 meter against "data storms."
    • In this case it was a rogue device pumping out too much garbled date. The solution being to design nodes that isolate the network from such occurances. The devices communicate through some high level protocol that is validated by the nodes before getting on the network, not as seems to be in this case standard TCP/IP over Ethernet. The loss of two recirculation pumps because of a network event is not trivial.

      Network stack has too high priority (Score:1)
  • In the letter from the Committe on Homeland security to Nookoolur Regulatory Commission about it,
    In accord with current regulations, NRC staff decided against investigating the failure as a "cybersecurity incident" because 1) the failing system was a "non-safety" system rather than a "safety" system,

    If failure of the component is dangerous enough to force a system shutdown due to lack of cooling, that IS a safety system. It's like saying there's no need for bracing the BOTTOM of the ladder, cuz it's
  • by maz2331 ( 1104901 ) on Sunday May 20, 2007 @11:36PM (#19204239)
    A "data storm" can be caused by lots of things, even an unstable driver causing a NIC to spew garbage packets. Or an application that hits a bug and begins spewing to the network. Or a failure of Spanning Tree causing network loops to arise (which can really mess up an Ethernet).

    The wierdest I ever saw was a situation at a school where the entire network (built around high-end Cisco switches) crashed hard. It took 3 hours of troubleshooting and disconnecting various segments to finally pin down the cause. It was a little mini-switch that some teacher attached to the LAN that somehow had a meltdown and began spewing "valid" Ethernet packets with all kinds of random garbage source and destination MAC addresses, random payload, and valid checksums. No hosts were attached to the mini switch, so it had to be something in its microcontroller going haywire. This cause every switch to go nuts trying to maintain its forwarding tables ("show cpu" was 100% utilization) and resulted in no traffic going anywhere. It even crossed VLAN boundaries since all the switches had "trunk" ports using tagged VLANS, so the garbage packets still made it through the entire LAN.

    These things happen sometimes. Network gear is generally pretty robust, but can still fail in wierd ways.

It appears that PL/I (and its dialects) is, or will be, the most widely used higher level language for systems programming. -- J. Sammet

Working...