Blackout Cause: Buggy Code 377
blanca writes "The big northeast blackout from last summer was caused in part by a software bug in an energy managment system sold by General Electic, according to a story on SecurityFocus. The bug meant that a computerized alarm that should have been triggered never went off, hindering FirstEnergy's response to the train of events that lead to the cascading blackout. Investigators found the bug in a intensive code audit following the outage, and a patch is now available."
fp? (Score:4, Funny)
Well, that statement is only half false, it's reliability has been field-proven.
More Reliable than Mars Rover (Score:5, Insightful)
The Mars Rover's software crashed in just a few days.
Virtually all software should be designed and tested better than it is.
However, I'm perplexed at why the Mars Rover failure and resurrection is considered a miracle of human inginuity, rather than an indictment of crummy testing.
I'll not excuse the power grid software either; but it seems to work more reliably than the software on the Rover.
Re:More Reliable than Mars Rover (Score:3, Insightful)
It is not considered a miracle but it is considered amazing. It is hard enough to debug things sitting on your desk, harder to debug someone else's problem over the phone and worse from orbit but imagine debugging a problem with 10 minutes of light delay! And there is only one computer on that rover so they were using the buggy computer to recover; not an easy task. In the end it turned out to be flawed file management code in the flash memory; the daily TODO list was kept in flash and it couldn't find i
Re:More Reliable than Mars Rover (Score:4, Insightful)
Complete testing is impossible.
Re:More Reliable than Mars Rover (Score:5, Insightful)
"Software sucks because users demand it to."
Unless every single software company does this, the ones that don't will own the market by virtue of supplying software that "mostly works" two years ahead of the others that supply software that is "perfect, minus epsilon". Then, all of the perfectionados go out of business, and the market returns to its present state. Things are the way they are because that's how various market pressures make them.
Re:More Reliable than Mars Rover (Score:4, Insightful)
The market is slowly changing, thankfully. A good example of a maturing market would our good old friend: home electrical wiring. How long did it take before every new home since probably the early 1980s is wired pretty much identically. They went through several different types of wire and insulation, grounded and ungrounded outlets, fuses and circuit breakers, etc. In a lot of ways, the software world is no different, and I'd say were at the aluminum wire stage with the various incarnations of systems we have and accompanying reliability and security problems.
Re:More Reliable than Mars Rover (Score:4, Informative)
Re:More Reliable than Mars Rover (Score:3, Informative)
What you should be asking is why is it so difficult to write bug free code? The obvious answer is because developing and testing code is harder than you realize. A simple if statement looping 10 times will have over 1000 different code paths that you would need to test if you wanted to be thorough. So a large software project makes this kind of testing impossible.
What people try to do instead is use Pared
Yeah, right. (Score:5, Funny)
Re:Yeah, right. (Score:5, Informative)
>>nope, MV... though it may have been 45MV...
The first guy is right; there is no such thing as a 45 MV transmission line. The highest voltage transmission line classification is 765 kV. (That would be 0.765 MV.) In the mid-1970s American Electric Power and Ohio Brass played with some experimental 1.5 MV transmission equipment but they killed the project when they realized land owners would never let AEP put a 1.5MV line in their back yards.
The lines that First Energy put in the trees were 345 kV. I'm guessing they were rated to carry between 1000 to 1500 MVA. I have no idea where the 45 number came from or what unit would have been associated with it.
--zawada
Re:50MV arc'd to a tree (Score:5, Interesting)
The land beneath the lines was clear-cut about 12 years ago. But there are now trees under this line that are about 10 meters high.
Years ago when my wife was concerned about "power line emissions" the power company loaned her a meter that showed "electrical fields." I don't remember the scale, or even what it was supposed to measure, but I do remember that we had to actually get about 200 feet from the wire before the field from the line stopped affecting the meter. (Yes, on a humid summer day I once stood in my back yard with a neon bulb and caused it to illuminate by simply dangling a three foot wire from one lead and touching the other.) I had always assumed it was a 750kV line, and that the 100 foot easement was more than sufficient. Now, I wonder. Hey, maybe this is enough of an excuse to go out and get one of those IKE toys!
Re:Electrical Field Exposure? (Score:5, Informative)
How about electric blankets or heating pads? How about a battery powered shaver?
You expose yourself to these fields every day to an extent far greater than what you may have received from that transmission line.
By the way, you can light a neon light with a bit of wire and very little power. You can also light it with a MW AM broadcast transmitter less than a mile away; you can light it with a CB radio; and with just a bit more wire, and a location closer to the poles of the earth, you can light it when the earth is hit by a solar flare. Many among the various eco-scare-monger groups like to make this demonstration as if it were an indicator of something dangerous. If it were, there would be no life anywhere near the Arctic Circle.
Aside of the poor maintainance for the clear-cut area, you really have no need to be concerned about this.
Not very analogous... (Score:3, Interesting)
The phone may generate more relative power, but it's at a different frequency- in regards to electricity and the human body, frequency matters as much as anything else.
For DC, 10ma of current may not be noticable to a person.
For 50/60Hz AC, it's going to
Re:Not very analogous... (Score:3, Insightful)
It gets worse (oh, and not 50 MV) (Score:3, Informative)
Then, just when you most need the power, a tree that used to be at a just barely safe distance shorts the power line.
The high end for mainstream deployments, by the way, is 750 KV or 1 MV. Corona losses get really bad above that level.
Uh... (Score:5, Interesting)
Re:Uh... (Score:5, Informative)
Re:Uh... (Score:5, Insightful)
Even so, there should have been sufficient watchdog messages between the client, the server, and the field hardware for the XA/21 to broadcast a general alarm along the lines of "I can't talk to the stinking field, so we're all flying blind here, you morons!" This is exactly the same as software in my industry (HVAC fire/security systems for large buildings), where if you lose communication to a subsystem or the field, you have to raise alarms all over the place.
The real question is how you could lose such comm and the operators had no visible indication that they were relying on old data. This sounds like a missed requirement, if not insufficient testing.
Tim
Re:What does the watchdog watch? (Score:4, Interesting)
The alarm subsystem is often a seperate process. It doesn't talk to the field. That's the job for other elements of the SCADA system. It was supposed to watch for semaphores, messages, or read shared memory somewhere. How do you watchdog something like that if it gets the message, but doesn't do what it's supposed to?
In a SCADA system near and dear to my career, we set alarm thresholds so low that the operators expect a certain amount of alarm traffic even for routine events. This helps to discover any misbehavior in the alarm system.
There is such a thing as a control center which is TOO quiet.
It's dark here (Score:2, Funny)
Patch Available (Score:3, Funny)
Phew! then at least i can patch my own power craft before anything happens!
Re:Patch Available (Score:3, Informative)
This spells trouble (Score:3, Funny)
Re:This spells trouble (Score:5, Funny)
Subscribe to Slashdot -- we have to keep these guys employed and out of the real world!
Wrong article! (Score:5, Funny)
Yes but how is Microsoft responsible? (Score:2, Funny)
Re:Yes but how is Microsoft responsible? (Score:3, Funny)
Re:Yes but how is Microsoft responsible? (Score:2, Interesting)
Re: ms WAS responsible - chain of events (Score:5, Funny)
How's that?
the bug of my dreams (Score:5, Funny)
Oh good... (Score:2, Funny)
Where's the URL, dude? I want to apply it to my local copy.
See what happens? (Score:4, Insightful)
I've said enough.
speaking of outsourcing... (Score:3, Interesting)
With all the lip service about "homeland security," one ought to be concerned about anything affecting national infrastructure being sent abroad where you really don't know who is doing the coding, whether the coding projects are being further outsorced to say alQaidaSoft, etc.
Re:speaking of outsourcing... (Score:5, Informative)
Re:speaking of outsourcing... (Score:4, Informative)
Another opinion: maybe Blaster is to blame (Score:3, Interesting)
http://www.schneier.com/crypto-gram-0312.html#1 [schneier.com]
A snippet of the article:
Argument from ignorance (Score:3, Insightful)
Yeah, well I don't know that I won't be fired tomorrow for reading Slashdot at work, but that doesn't mean that I will.
And Another... (Score:3, Interesting)
Certainly, the energy corporations must be somewhat culpable for not rigorously testing the software in the first place? It is not in the interest of a for-profit company to see to it that such systems are functioning correctly, as that cost will detract from the bottom line profit. Only when disaster strikes can they be goaded into looking into problems.
Re:Another opinion: maybe Blaster is to blame (Score:4, Informative)
In short, the Microsoft bashers were wrong -- and at least Security Focus had the guts to acknowledge it.
Development vs Engineering (Score:5, Insightful)
I am a software developer not an engineer, as are most people in the field. Software won't become an engineering science until companies are willing to pay for that process. Given the current trend towards cost cutting I don't see that happening anytime soon.
Re:Development vs Engineering (Score:5, Interesting)
Is it true that some states have prohibited Microsoft from issuing MSCEs? I heard this somewhere but I can't remember. Something about Microsoft not having the authority to certify engineers.
Re:Development vs Engineering (Score:5, Funny)
But couldn't the "Microsoft Certified" part be interpretted as a disclaimer? Something along the lines of "Burger King Certified Brain Surgeon".
Re:Development vs Engineering (Score:5, Informative)
There is only one university in Canada that is actually allowed to graduate "Software Engineers," and it's in Newfoundland (MUN). Other universities are not allowed to call their grads "Engineers" unless they follow the strict cirriculum requirements of the main engineering authority in Canada, whose name escapes me at the moment.
This is all second-hand info, spoken as a guy who's married to a genuine, certified Engineer (Industrial).
Re:Development vs Engineering (Score:5, Interesting)
Doctor is not a protected term. Perhaps you mean "Medical Doctor"? There are lots of non-medical doctors.
I was arguing once with a MD friend of mine who thought that PhDs (like myself) don't have the right to call themselves Doctor. I explained that while medicince has been around for a very long time, the degree of MD has not. PhDs degrees have a much longer history than MD degrees.
It gets very funny when another friend of mine (who has a PhD in nursing) is called "Dr" in her hospital.
Re:Development vs Engineering (Score:4, Funny)
Heh, that reminds me of a friend of mine [unclekage.com] who happens to be a PhD. He likes to poke fun at MDs by saying, "Back in the middle ages, it was the learned scholar who was called 'Doctor'. The man who cut into you was called 'BARBER'!"
And he's teased his physician about this on several occaisions, saying things like, "Just take a little off the top, please!".
Re:Development vs Engineering (Score:4, Informative)
Furthermore, each province has a regulatory body which manages licensing of Professional Engineers (P.Eng.'s) which is a regulated designation. In Ontario this body is the PEO [peo.on.ca]. They have a webpage here [peo.on.ca] on the whole "software engineering" issue.
Re: Development vs Engineering (Score:5, Insightful)
> I am a software developer not an engineer, as are most people in the field. Software won't become an engineering science until companies are willing to pay for that process. Given the current trend towards cost cutting I don't see that happening anytime soon.
It will be interesting to follow the lawsuit news on this one. If someone gets squeezed hard enough, we might see a movement toward good engineering praxis as a result.
More likely the politicians will step in and bail them out, but ISTM that as society continues to rely more and more on software, at some point we're going to decide that we can't afford not to set and follow good engineering standards.
Re:Development vs Engineering (Score:3, Informative)
I've actually concluded myself that software development _can never_ become an engineering discipline, it's too creative a process for that. A software developer is more an artist than an engineer.
Really.
Re:Development vs Engineering (Score:5, Interesting)
You have 4 main variables in the software development equasion: Time, Quality, Functionality, and Efficiency. Notice that we only measure time, not man-hours or monetarycost. As we know from reading The Mythical Man-Month [amazon.com], we cannot reduce time by adding more people or by spending more money. While we list efficiency as a variable, we really have to treat it as a constant within the scope of a single release cycle. Improvements in efficency are generally very gradual and incremental, and for the most part cannot be effectively implemented in the middle of a release cycle.
I postulate that Time is directly proportional to the product of Quality, Functionality, and Efficiency [T = EQF]. Since E is constant within the scope of a single release, we can't use process improvements or similar techniques to improve quality in the short term. Assuming our goal is to improve quality, we either have to decrease functionality or increase time. Since monetary cost is directly proportional to time (time is money!), managers are very reluctant to give you more time. Furthermore, we are frequently under hard time constraints due to contractual obligations or market pressure. If we can't change time, we either have to sacrifice quality or functionality. Missing functionality is very obvious, whereas low quality isn't necessarily noticable in the short term, so it should be no suprise that quality is almost always takes the back seat to functionality.
Re:Development vs Engineering (Score:3, Funny)
When I was young and dumb, I thought it was neat to have "Software Engineer" on my business cards. After a few years of seeing just how inept/underfunded/constrained nearly all software developers are, I changed my job title. Calling a typical programmer a "Software Engineer" is sort of like calling a convict in prison a "Legal Countermeasures Engineer."
Would this be any better in an OSS environment? (Score:4, Insightful)
Who thinks this could have been any better with Open Source and why?
People make the comment of the many eyes, but who is really looking at the code?
Re:Would this be any better in an OSS environment? (Score:2)
though maybe they could have used proven building blocks for other parts from os and then focused on the parts they had to do, though they might have done this anyways.
what's stupid is that the whole blackout cascaded to a such large area. like, there shouldn't have been a possibility of that even if the software had been intentionally flawed..
Re:Would this be any better in an OSS environment? (Score:2, Insightful)
Re:Would this be any better in an OSS environment? (Score:5, Insightful)
Open source almost certainly would have not prevented the bug. The bug might have been found faster after it happened though, because curious (or under pressure from their boss) engineers engineers in every facility affected would spend at least some time trying to figure out what went wrong.
Having the source is great, and you would be surprised at the number of companies who license the source for what they use. Risk management is important. Free isn't everything, you can get many of the same things by paying
Re:Would this be any better in an OSS environment? (Score:4, Insightful)
There are also system reliability requirements to be considered. Hardware fails. Software fails. Is the system designed to detect and cope with component failures?
GE's software may suck. I don't know. I've never seen it. I am suspicious of people who attempt to hide their own negligence by blaming a third party.
Blame Canada (Score:3, Funny)
Re:Blame Canada (Score:2)
It was the Canadian arm of GE
The programmer was Canadian
See - it still works!
Re:Blame Canada (Score:2)
Perhaps this project was outsourced to India? Wouldn't it be lovely if we could bash Indians and Ohio in one article?
Bad bugs (Score:5, Informative)
Re: Bad bugs (Score:2)
> Chalk up another one for the most disasterous software bugs in history. This one should give the Ariane 5 explosion a go for no 1.
The A5 wasn't caused by a bug, at least not in the sense we usually use the term. It was caused by a decision to re-use a part from the A4 and its embedded software, without bothering to review its specifications.
It's certainly a problem that good "engineering" should have caught, but most of us wouldn't call it a bug.
Will they apply it?! (Score:5, Funny)
I'm waiting for the next big power failure, then the excuses about why the patch was never applied. :)
Re:Will they apply it?! (Score:2, Funny)
Hmmm... (Score:4, Funny)
One coder to code it,
One debugger to miss the bug
and into the darkness lead them.
way, way off-topic ... (Score:2, Funny)
Sound zee alarm!!
Software "Engineering"? (Score:5, Insightful)
A look at the software industry will show this to be the norm. And that is why there is such a problem with having people claiming the title of "software engineer". "Engineer" doesn't just mean having the technical savvy, it also means having a responsibility to the public for the use of that knowledge and being beholden to a professional body charged with ensuring you are held accountable.
Re:Software "Engineering"? (Score:5, Insightful)
Re:Software "Engineering"? (Score:5, Insightful)
How long do you think engineering (as it stands today) would last if that bridge meant to stand on bedrock spanning no more than 1000' and carry a load of no more than 1500 tons at any given time were suddenly put on a sandy bed, stretched to cover 1100' and carry 1600 tons... oh yes, and the user didn't like that third support so they removed it.
Software and engineering are VASTLY different disciplines. If software is ever judged like engineering then it would kill the market because the EULAs would have to say that you use THIS motherboard with X amount of RAM and Y amount of hard-drive space. The agreement would only be in effect as long as you used OS "ABC" and no other processes besides those required by the OS and the programme in question were running. It would make the cost of running a business prohibitively expensive.
When you consider that most large-scale software development projects are equivalent in complexity to building structures like the Golden Gate Bridge or the Empire State Building (I didn't want to mention any buildings outside the US since I realise the audience on here is largely American and probably wouldn't know what I was talking about) consider the cost of actually treating software development the same way... I'm sure companies everywhere will be lining up to pay $300M for that content management system.
Re:Software "Engineering"? (Score:5, Insightful)
Currently, software is built in a craft/guild model: senior developers (masters) teach junior developers (journeymen) who've reached a certain level of expertise. Interns (apprentices) are drafted into the profession and groomed into junior devs. There is a widely held notion of subjective quality, and we can recognize a masterwork, but we can't quantify what it takes to generate one.
Software engineering will become a true engineering discipline only when there is an objective measure of defect level and an objective notion of what constitutes an adequately circumscribed operating environment. Once we have adequate definitions of those things, though, software production will become industrialized almost immediately.
Re:Software "Engineering"? (Score:5, Insightful)
Software Enginners would have to carry E&O insurance (Think of it as malpractice insurance, like a MDs). It MIGHT be supplied by their boss, but...
And in exchange for taking on this risk, what would a software Engineer EARN? You'd better believe it would be a LOT more than it is now.
You would still have "coders" - in fact, MOST "software engineers" would go back to their pre title inflation title - "Programmer". The SE on the job would be responsible for all the code that the programmers wrote
Just like MOST jobs don't have to be signed of by a PE, most software would NOT have to be signed off by an SE - but if you use software that wasn't signed off by a SE, and you caused 50b in losses, you would loose YOUR shirt
At this point in time, it seems that the people of the US just have NOT found the need to come up with the idea of a licensed SE. I predict it will happen, and within the next 25-30 years. There have been movements withing the programming trade to do this. it's coming - but when?
Right now, software development is very much like the "guilds" of the Middle Ages. You didn't have PEs back then - you had folks who learned from other folks, and you had projects that failed massively. Eventually, things became codified, and a lot of the failures stopped - at least for day to day stuff. But guess what? Buildings still fall down, even in construction (read the book "why buildings fall down"). It's just that for "common" designs, it doesn't happen
Re:Software "Engineering"? (Score:4, Interesting)
Creating a true software engineer is different than making them PE's. Right now, most of the engineers that design things in industry don't have PE's and if they do, they don't make it known publicly for the very reasons you mentioned.
The rest of us with out PE's don't need the insurance, as that is supplied by the company.
Also, keep in mind that just because an engineer worked on something doesn't mean that it will be expensive. Most of what I engineer costs less than a dollar.
If you haven't guessed, IAAE (I am an Engineer)
Re:Software "Engineering"? (Score:3, Informative)
SEI level 4 and 5 shops.
So if tougher standards are required, more work could go to India.
The required activities to get to SEI level 3 are mostly management, so programmers by themselves cannot bring the level of software development beyond that.
What about the actual Engineers involved? (Score:5, Interesting)
What about a good Electrical/Mechanical/Civil Engineering solution that would have prevented it from cascading through different systems / electrical companies / countries?
One piece of software which didn't raise an alarm is shocking. The fact that it cascaded over such a wide area is simply mind blowing.
Before we talk about "software engineers" how about talking about "traditional engineers" and their role in this massive failure?
Typo... (Score:3, Funny)
This is Slashdot! Isn't that supposed to say Microsoft? It's always Microsoft.
Who coded this? Homer Simpson? (Score:4, Interesting)
When a backup server kicked-in, it also failed, unable to handle the accumulation of unprocessed events that had queued up since the main system's failure. Because the system failed silently, FirstEnergy's operators were unaware for over an hour that they were looking at outdated information on the status of their portion of the power grid, according to the November report.
How in the world did they manage to build a system nearly completely dependant upon computers, and yet not know when they lost not just one, but two computers that monitored the system?
Homer: Don't turn off the computer! Don't turn off the computer! Don't turn off the computer!
"Click"
TIBCO middleware (Score:3, Insightful)
And yep, it runs on major critical systems, including energy systems and satellites.
Lean on it in the slightest and it will crash and burn with little chance for recovery. Tibco even says they don't test their own software (lack of docs lowers their liability). Press them for test results and they will offer you to pay them to test for you.
When a backup server kicked-in, it also failed, unable to handle the accumulation of unprocessed events that had queued up since the main system's failure.
Sounds like classic Tibco.
Did we steal this code from the Russians? (Score:2, Funny)
Metroid (Score:5, Insightful)
The grid in the North East US is supplied by horribly inefficient and antiquated power lines that were struggling to keep up thirty years ago. That they are still in use today is an outright crime. There's also the issue of the operators of the lines generators trying to save a few bucks by cutting maintenance on equipment and facilities and cutting supervising staffs down to skeleton crews. It is much easier to fit "software bug" into a sound bite so the news media will stick with that. Unfortunately the real cause of the black out is not ever going to be patched and another blackout is as inevitable as this last one was. I hope next time a few more people will have invested in backup generators or some alternate form of power to keep from losing their business during a blackout.
Re:Metroid (Score:3, Insightful)
Wow. Quite an accusation. Any facts to back it up?
Really? There are major circuit outages on the Eastern Interconnected Network every day. The system is designed to have the local area go black instead of blacking out a widespread area. That was the lesson of
OK, time to revisit advanced development methods (Score:3, Insightful)
Yes, they're difficult. Yes, they aren't likely to eliminate all bugs. BUT. They provide a much better chance (as I understand it - I'm not an expert) that what is designed is what actually gets implimented. That shifts the burden onto the design, but that's OK - that burden was always there. It just means that the design gets properly implimented, which is all that can reasonably be asked of the coding process.
Currently, again as I understand it, the life of a software program in development is a constant struggle by the developers to cope with ever changing demands of customers. I think if people want matters to improve the customers are going to have to come to grips with reality, take the time to sit down and think things through, and make all critical design decisions BEFORE the development process begins. More expensive up front? You bet. That's why I think companies should look at cooperative effort for this type of thing. Distribute the cost of developing one really good program across an industry. A lot of the same core functionality can likely be shared between businesses - if they all pay for one proper design and implimentation of an open program up front, and they all get copies of the logic and proof code with rights to extend as they see fit, they all benefit. They can also open up the more general parts of the package to the world at large under GPL, and anyone could contribute who can generate valid B and Z designs/proofs. Sort of an "academic" open source code development forum - peer review and all. The companies get the benefit of all new development - if they are using it internally they can extend the GPL code for themselves, so long as they don't distribute it. If they do distribute it, they can so so under GPL for everyone to enhance. A plugin based model can also allow them to develop components to the system they can sell as commercial software, if they wish.
Whether this would work/appeal with corporate thinking I have no idea - many of those folks seem to view cooperation like the plague. But it might allow a higher grade of software to be developed and universally used, and I have a hard time imagining how that could be a bad thing for anyone.
Not Surprised (Score:4, Insightful)
Given my personal experience with this certain Fortune 5 company and software development as a whole, I am not surprised.
The bottom line is that there is soooo much software developed here by non-computer programmers. There are many great Engineers (Mechanical, Aerospace, etc.) here, yet very few can write good code. Many of them are asked to write code nonetheless and thanks to the travesty that is Visual Basic and other Rapid Application Development tools the code that is produced is extremely un-maintainable.
Then you have the matter of people moving jobs every 2 years and the poor bastard who has to maintain someone else's code gets lost inside of it.
Consider me very frustrated at the whole process.
SCADA is really neato... (Score:2, Informative)
No-one writes flawless code, not Sun, not IBM, and not even Linus or Alan Cox or Larry Wall. Anything that is controlled by code is bound to break, but that
Argument against centralization (Score:4, Insightful)
To me, this report give a good example of why a monolithic (monocultural) dispatching system is not a good idea. If every transaction were controlled by a central center, a single software bug could shut down the entire North American grid.
sPh
Re:Argument against centralization (Score:3, Funny)
This may seem impossible to people living in today's world, but it makes perfect sense in a world where technology is so efficient and perfected that every household can easily af
Apparently, not DCOM/OPC related (Score:3, Informative)
Interestingly enough, the sales literature describes it as having, "[an] established track record of field performance - over one million hours of online operation."
I wonder if they'll revise the brochure now?
Tim
We kick MS, but GE did the wopper... (Score:3, Interesting)
Software engineering *not* possible. (Score:4, Interesting)
Furthermore, the tools we have for the job are inadequate. The programming languages are primitive. The debugging tools are dumb. The machines are not clever and strong enough to prove the mathematical theorems behind its program. We don't even learn these things in college...we learn how to use programming languages, but we don't learn how to program...but I seriously believe we will never learn how to program, because a program's complexity increases tenfold for each line of code written!!!
Try something new (Score:3, Informative)
blaming the software is easy (Score:3, Insightful)
So the software didn't raise alarms as it should've. That's bad. But it seems to me that the software is being made a scape goat here. It's much easier to blame "that #$@&@$ computer" than "FirstEnergy's failure to trim back trees encroaching on high-voltage power lines" or the fact that the infrastructure for the powergrid is old and poorly setup such that one failure can bring down the whole system. There's no reason why a failure in Ohio should blackout New York and there's nothing software can do to fix that.
No Wintel bashing? Oh wait it's RISC/UNIX code! (Score:4, Insightful)
Funny, just because this ships for "industrial strength" AIX / Solaris RISC systems (see specs on pg 8) [gepower.com], I don't see any cheap, reflexive comments about the platform.
I guess the message here is that good or bad code can be written for any architecture.
The alarm bug contributed but was not the cause (Score:5, Informative)
The SCADA system itself did not fail, but its alarm function did, which provides alarms to control room operators about system operational problems. The problem with the alarm function seems to be a case of too many alarms for the system to handle as the problems multiplied. The software bug that they are now reporting was probably related to the unexpectedly large number of alarms that the system was experiencing. The new alarm inputs built up and then overflowed the process input buffers. The alarm system just stalled while processing an alarm event and the alarm function stopped. Then, at 14:41 the primary server hosting the alarm processing application failed due to some combination of the stalling of the alarm application and the queueing to the remote terminals. The hapless backup server then was automatically activated and everything was was transferred to it, even the functional non-alarm stuff. The backup server failed after 13 minutes. Basically, the SCADA alarm system seems to have been massively overloaded (which shouldn't ever happen, of course) beyond the capability of the system design to cope with. The bug apparently prevented an indication that the alarm system was failing but it looks like the cascading failure still would have occurred even if the software bug had not been present because the system deterioration had progressed to far to recover by the time that the bug manifested itself.
The immediate cause of the failure seems to be the forgetfulness of the analyst who was operating the planning model. The significant underlying contributory cause seems to be a very poor regional operational design in which a critical centralized system planning tool was being used with insufficient backup and oversight. It looks as though both Unix and Windows escape blame. The SCADA system probably was doing far more than it's designers intended and probably performed heroically until it died. 'Aye Captain...I canna do no more.'
We win again! (Score:3, Funny)
Software: 2
Hardware: 0
Re:Hmm (Score:3, Funny)
Who knows, perhaps it was only the overhead lines that went dead
Re:Hmm (Score:5, Interesting)
I bet they had much wider safety margins built into the system which prevented blackouts. But these safety margins probably cost money ( I say this without knowing a thing about the electrical system ) they probably mean a less efficient use of resources. So power companies buy GE's software. They don't buy it so that they can have an added measure of blackout prevention, they buy it because it enables them to cut out expensive/inefficient safety margins without (supposedly) sacrificing reliability. They do this to lower their cost of providing electricity to you.
Re:GE Outsourcing To India (Score:5, Informative)
facts before hysteria thanks
Re:GE Outsourcing To India (Score:4, Funny)
It is unpatriotic to move them from California, where they belong! I bet they pay the people in Florida a lot less.
(This is a joke)
Re:This is unacceptable (Score:5, Informative)
Re:Text of the article (Score:5, Funny)
Re:Text of the article (Score:2)
Re:A patch is now available (Score:2, Informative)
By the way, the actual bug... (Score:3, Informative)
"As it happened, the problem itself - the problem per se - took this form. A piece of telco software had been written in C language, a standard language of the telco field. Within the C software was a long "do... while" construct. The "do... while" construct contained a "switch" statement. The "switch" statement contained an "if" clause. The "if" clause contained a "break." The "break" wa