NERC Releases Interim Report on Aug 14th Blackout 426
will writes "The North American Electric Reliability
Council has released four documents concerning the
August 14th power outage power outage in the North East. The blackout
investigation homepage lists all NERC's documents relating to this
event. Press coverage is at The
Washington Post, CNN,
and CBS
News. The take home message: FirstEnergy
did it. The are, of course, denying
it." The report is also available at reports.energy.gov. Reader stinkydog writes "According to Yahoo News part of the blame for the big fizzle of 2003 lies with a failing SCADA system, GE's XA/21 power management system. 'Not only did the software that controls audible and visual alarms stop working at 2:14 p.m. EDT, but about a half hour later, two servers supporting the emergency system failed, too.' According to the product specs, it is a Unix system with X Windows."
Go get 'em! (Score:5, Funny)
"According to the product specs, it is a Unix system with X Windows."
Good, let's sue SCO!
ps: f1st pr05t
Re:Go get 'em! (Score:5, Funny)
Darl McBride had repossessed the system, leaving a bill for $699 and a subpoena in its place.
Re:Go get 'em! (Score:5, Funny)
The reason they use X is because their systems are managed by MCSEs.
No, Sue the End User (or at least the analyst)! (Score:5, Interesting)
heh!
Alas, as Microsoft apologists are wont to point out (even in the many cases where the crash or security flaw doesn't stem from mismanagement or configuration errors), a misconfigured system ins't going to work regardless of what OS it is running.
In this particular case, quoth the article
The technician forgot to restart the monitoring software. Oops.
Following in $CO's illustrious footsteps, I think perhaps it is time we sued the poor schmuck who forgot to restart the monitoring program. Or better yet, the company dumb enough to hire him, the electrical company. After all, according to Our Lord and Master Darl McBride, End Users should pay (and pay heavilly).
Re:No, Sue the End User (or at least the analyst)! (Score:4, Interesting)
Seems to me the stupidity predates the technician.
Re:Go get 'em! (Score:2)
Not SCO.
Oh boy, here it comes (Score:2, Funny)
Send in..... the Trolls.
Are they into warez? (Score:2, Funny)
just wondering if their webmasters are into the warez business...
X Windows? (Score:2, Informative)
Re:X Windows? (Score:2)
Re:X Windows? (Score:2)
As Winston Churchill might have said, "You are a pedant up with whom I cannot put."
Re:X Windows? (Score:2)
Flame bait (Score:2, Funny)
Re:Flame bait (Score:2)
Even the most stable UNIX systems are vulnerable to hardware failures.
Re:Flame bait (Score:2)
Told Ya So (Score:2, Funny)
Microsoft sucks (Score:5, Funny)
You shouldn't use MS products for such a critical system.
What? It was a unix system?
Must have been a hardware failure then.
[/slashdot]
Re:Microsoft sucks (Score:2)
Re:Microsoft sucks (Score:2)
The reason the DNS servers are super-reliable is because there are many of them, in a federated architecture. 1 can crash hourly, and the web will still work.
Re:Microsoft sucks (Score:2)
Memo to power company: (Score:5, Funny)
Memo to power company:
Put power-system controlling servers on UPSes :-)
(yeah yeah, I know, it wasn't because they lost power. Its a joke :-)
about a half hour later (Score:2)
(probably not the true scenario, but still stupid enough to be possible)
Now we know... (Score:4, Insightful)
Flame me if you want but France is known to have one of the best electrical system in the world. The government owns it [www.edf.fr] and some laws do protect the costumer. It allows France to export a lot of power to other European countries thus making a lot of money from it and playing a big role in Europ energy.
I think France made good choices for power:
- nuclear (or nuculear whatever). Much cheaper and cleaner than oil. France is very strong on nuclear technology and do export its knowledge to other countries. To date we didn't have any major incident and I think it's pretty secure as long as enough money is spent.
- public service. There is a law in France stating that every citizen, wherever he lives, as the right to have access to electricity for free. Of course we pay bills but if you live in the country the government MUST bring you access to electricity even if you live far from everything. It's a law so sometimes it's not really followed but most of the time the government sticks to it.
When I arrived in the USA I was shocked by the poor quality of your electric system. Many outages, expensive bills and thousand of wires in the sky ! I think that it's a very advanced country with an outdated power system. I've seen on the TV recently that many companies as starting to produce their own power. I really think it's pretty bad since the installations needed are dangerous, potentially explosive and very expensive.
What happened on August the 14th also suggests that a country really depends on its energy distribution system and that it could be, in the near future, target for terrorists.
Re:Now we know... (Score:5, Insightful)
Re:Now we know... (Score:2)
Does this make sense?
It's deregulation (Score:5, Insightful)
I disagree completely -- until the US deregulated, we had an extremely reliable power system that was able to expand dramatically for many decades to keep up with demand while keeping prices low. Then some lunatics convinced enough people that it was a good idea to make power companies unregulated monopolies (i.e. they were no longer required to invest in maintenance and infrastructure, or to maintain excess capacity to cover emergencies or power spikes, and removed the cap on profit margin), after which the power companies did what you'd expect -- raised rates like crazy while slashing spending on infrastructure maintenance and expansion. The result is that a bunch of investors and CEO's made tons of money while the capacity and quality of service that they provided suffered. The problem was not just the big outage -- the number of instabilities in the power grid that didn't result in actual outages has been increasing dramatiaclly for decades.
See http://www.ncpa.org/iss/ene/2003/pd081503a.html for some more info.
Who do I blame? The politicians who allowed themselves to be bought off (or conned) by the power companies, to the detriment of the entire country.
Re:It's deregulation (Score:3, Informative)
Deregulation simply gave the utilities the opportunity to operate in three parts: Transmission, Generati
Re:Now we know... (Score:2)
Re:Now we know... (Score:4, Interesting)
The biggest difference is that France puts their power lines underground. Here in Silicon Valley, some French colleagues were shocked at the frequent power disruptions during the winter. Well, of course the power was out, the locals tell them, there was a big storm! (The "big storm" consists of an inch of rain and a short period of 40 mph winds, which knocks down all the power lines).
Re:Now we know... (Score:2)
That blackout should have been a terrifying wakeup call to everyone in the country. Had it been caused by an act of deliberate sabotage (a large bomb on that same place on the power grid would achieve the same effect) the hypothetical attackers would have had an entire weekend to pummel New England with virtually no way for the area to respond.
Re:Now we know... (Score:2)
And on top of those problems, every time we the people demand an upgrade the government would happily raise our taxes by twice whatever the upgrade would cost and then take twic
Re:Now we know... (Score:3, Interesting)
Our power grid has not been upgraded significantly in decades. The power companies have shown quite well that, without direct regulation, they are not going to willingly do so. (especially as the costs of upgrading go up every year technology advances and the grid goes up)
Re:Now we know... (Score:2)
Re:Now we know... (Score:2)
Re:Now we know... (Score:3, Funny)
Anyway, it's still unacceptable that such things happens in the most powerful country in the world.
What, Canada? Don't say such things, we're really very shy.
The government owns it and some laws do protect the costumer.
Wow that's some country, laws to protect costumers no less. What about mimes and jugglers?
When I arrived in the USA I was shocked by the poor quality of your electric system
Well, you're definitely French, I'll grant ya that.
Re:Now we know... (Score:2, Insightful)
I also feel that we should use some solar in a reasonable fashion. especially in cities where they are consistently hotter, because of all the concrete. Out in the burbs where I
Re:Now we know... (Score:2)
Yes, and your temperatures between Florida and France are so similar. How much did you run fans, AC, and other cooling technologies in Florida as compared to France? Heck, how many people died in France this summer -- a summer which had highs several degrees lower than those in Florida (much less Arizonia or New Me
Re:Now we know... (Score:3, Insightful)
Uh, no. Heating is considerably cheaper to do than cooling, even if you do both by electrical means. In most areas of the world where heating is a concern you don't do it electrically -- because burning gas is much more effective. It's also considerably easier to insulate against cold than it is against heat.
Since I arrived 6 months ago: none in Florida.
And exactly where do you live in Flori
Re:Now we know... (Score:2)
As opposed to the downtown US cities with the same problems?
made me laugh (Score:2)
Re:Now we know... (Score:2)
Show me where I lied in the above post. If you're just trying to make a point, make it without going down the "americans suck" path, because you have exactly *zero* say in what americans drive, eat, spit or fuck. Period. And I am honestly sick and tired of hearing people like you excercise their freedom of speech (largely made possible by America) to bash America, and then complain when someone calls them on their bullshit. Are you complaining? Or do you have somet
Population density viz of the Eastern USA (Score:2, Informative)
Kucinich was on top of First Energy (Score:2)
Disaster again (Score:5, Funny)
Re:Disaster again (Score:3, Funny)
Re:Disaster again (Score:2)
Re:Disaster again (Score:2)
There's one thing to have foreshadowing clues (Asimov, for example) and another to leave them scattered all over the place so that not only do you keep tripping over them, you eventually fall and get hurt. That's just bad writing.
Re:Disaster again (Score:2)
Yeh, it was, that's right...but I think the chaotic havoc was caused by the guy with the taste for snacks, and a little bit of corporate espionage. Speaking of it though, what the hell was the file manager they were running anyways?? I remember that it looked kind of neat but seemed really slow and unresponsive (I just know that my dinosaur parks' servers would be running in CLI mode only).
It's a classic case of human error, not computer.
Re:Disaster again (Score:2)
Re:Disaster again (Score:3, Funny)
It was a third-party-application issue, not an OS issue. Unix itself is not to blame.
Report vindicates First Energy (Score:3, Funny)
From CNN Money's link:
"
Re:Report vindicates First Energy (Score:3, Interesting)
Where's Microsoft? (Score:2)
Ugh.
Seriously, though, the failure of a SCADA system **SHOULD NOT** bring down the grid. This is just passing the buck.
Denies? not really (Score:2)
"We recognize that our computer system experienced problems that day, which we discussed publicly immediately following the outage."
Is followed a paragraph later by:
"By focusing its analysis on a few selected events, the conclusions the Task Force reached don't address the complexity an
Re:Denies? not really (Score:2)
1. The MSBlaster worm was widespread on that day, and was wreaking havoc on both the Internet and on corporate LANs when employees brought infected machines in to work and plugged them in behind the firewall.
2. These Unix systems are old, and are probably running on 10 mbps unswitched segments of the co
Re:Denies? not really (Score:2)
You can talk about black helicopters 'till you're blue in the face, but that doesn't change what's in the report. Do yo
The XA/21 Did Not Break Down (Score:3, Informative)
http://www.gepower.com/prod_serv/products/scada
On page 99 of the report the task force states specifically that,"although there were a number of worms and viruses impacting theInternet and Internet-connected systems and networks in North America before and during the outage, the SWG's preliminary analysis provides no indication that worm/virus activity had a significant effect on the power generation and delivery systems."In other words, the power generation and delivery systems (GE's XA/21) running Unix were not affected. SCADA, the alarm monitoring system, however, was affected. SCADA failedto transmit alarm conditions the the monitoring workstations because the Windows platform it was running on was under a denial of service attack. The net result is that the operators were not aware of serious trouble with the grid before it was to late.
Re:The XA/21 Did Not Break Down (Score:3, Interesting)
SCADA failed to transmit alarm conditions the the monitoring workstations because the Windows platform it was running on was under a denial of service attack.
Do you have a better link? (Score:2)
http://www.gepower.com/prod_serv/products/scada_s
Brings up a 404 on Ge's website.
Mod parent down... (Score:2)
There is also no evidence, nor is there any information suggesting, that viruses and worms prevalent across the internet at the time of the outage had any significant impact on power generation and delivery systems. SWG analysis to date has brought to light certain concerns with respect to: the possible failure of alarm software; links to control and data acquisition software; and the lack of a system or process for some operators to view adequately the status of electric
Re:Mod parent down... (Score:3, Informative)
If anyone has some tangible evidence to the contrary, I would sure like to hear it/read the quote... As far as reading this blog goes though, I'm not too interested in hearing people quote one line, and make a completely orthogonal conclusion.
That's what I was saying... The grandpare
What caused the boxes to crash? (Score:3, Interesting)
(There should have been better firewalling in place if so, of course.)
Re:What caused the boxes to crash? (Score:3, Insightful)
If we don't want it to happen again, yes we do. It was more than one box, which points to something more systemic than a random hardware glitch or failure.
According to the specs, it's a computer with an OS (Score:2)
According to further documents, this computer's "operating system" was running some kind of "software program".
C'mon guys, if it was running Windows and it BSoD'ed, then it might be worth mentioning. Other than that I don't see why we should be pointing out what OS it's running. The failure could have been caused by any piece of hardware, so
Power out again @ FirstEnergyCorp (Score:2, Funny)
Seems that their server went down. I guess this means the northeast will be experiencing another blackout.
Blackouts by ./ Must be a first
Doh! I knew it. (Score:2)
During a critical hour before a final chain reaction of power line outages began around 4 p.m. Eastern daylight saving time, FirstEnergy grid controllers were flying blind, unaware of computer and power line problems on their system, according to the investigation headed by U.S. Energy Department officials and their Canadian counterparts.
Anyone else picture Homer Simpson here?
Who cares about the OS or the provider? (Score:5, Insightful)
Any carefully managed OS (inluding Windows) can be stable and predictable.
Any badly managed OS (including Unix) WILL be unstable and unpredictable.
Got it? Good.
Now as far as pointing fingers at the guilty parties, understand that the infrastructure is really at fault far more than any individual company. Look at this from a broader perspective: One company was able to take down how wide of an area? The whole system is too fragile, too interdependent, and maintained too close to full capacity. Worse, there's absolutely no incentive for a company to maintain a large reserve capacity, since it costs a lot, and brings in no extra income.
But of course when it happens the next time (and make no mistake--it will), we'll have another commission to once again figure out the single company that broke a rotten and unstable infrastructure, instead of fixing the root problem.
Re:Who cares about the OS or the provider? (Score:2)
The only rule is that the box will be in my possession and I am the only one allowed physical access to the input devices and the box iteslf.
Any takers?
First Energy is a horrible corporation (Score:5, Informative)
FirstEnergy used to be CEI. . . a horrible company in Cleveland that allowed a nuclear core failure through negligence and then tried to bully the government of Cleveland into selling its municipal power supply to it thereby granting it monopoly status in the region.
Check out the story here:
http://kucinich.us/powertothepeople.htm
The presidential candidate Dennis Kucinich was Mayor of Cleveland at the time and endured a browbeating when he refused to sell the Municipal power company to these scoundrels.
I'm not necessarily pro-Kucinich, but I am VERY anti-FirstEnergy. They have a track record of irresponsibility and dishonesty that should be enough to have the government dismantle them.
SCADA system users... (Score:2)
Usually these systems were put together by some integrator consultant years ago. The consultant goes on to bigger and better jobs, but the system remains pretty much as it had been set up.
The problem, as we gentle slashdot readers know too well, is that nearly all computers need security patches or updates of some sort applied. In addition, there are often configuration changes, log files which need to be looked at, and a thousand ot
What were the failed systems running? (Score:3, Interesting)
Power outage traced to dim bulb in White House (Score:5, Interesting)
FDR enacted regulation of certain industries to insure that consumers would not get ripped-off. Bush reversed these regulations possibly because he doesn't know history, and/or he and/or friends/relatives have stock in these industries, or some other overlooked reason. Bush and his administration heavily contributed to the power outage, and is making sure that plenty more are bound to happen. Remember this on the next election day.
Re:Power outage traced to dim bulb in White House (Score:3, Funny)
And whoosh! All of the good points Mr. Palast raised no longer apply.
W32.Blaster links? (Score:2)
SCADA Definition (Score:5, Interesting)
Supervisory Control And Data Acquisition
It defines nothing about whether or not COM, DCOM, OPC (Ole for Process Control), or any other proprietary communication framework is used (contrary to some other highly moderated statements you are likely to see in this discussion)
The bulk of serious SCADA systems in place are probably legacy systems of some kind, including many variants of UNIX systems (we have old micro VAX systems still in use). Many of the newer systems are Windows based, and are obviously subject the standard Windows viri, worms, etc.
Worse yet, these systems are very difficult to upgrade or patch, due to the critical nature of their duties. It is not unlikely that a large portion of the Windows based SCADA systems in the world remain unpatched, and are "safely" firewalled off from the internet.
Of course, the problem is that much of the monitoring gear used to diagnose network issues is also Windows based, and carries with it the standard retinue of Windows viruses and worms, right into the heart of the control center.
These UNIX systems have run (and will contiue to run) uninterrupted for years at a time. We have calendar alerts in place to tell us to go manually reboot unix machines after months of uptime, just to ensure that their SCSI drives will spin back up (in case of a control center power outage, etc.)
Somehow, I don't think that is an issue for some vendor's SCADA systems based on more popular OS's, but I might be wrong...
Not UNIX, But client - Server (Score:2)
Blame Canada (Score:3, Insightful)
Cleveland eye witness to FirstEnergy (Score:3, Interesting)
The day of the black out, I was heading home to my house in Cleveland. Naturally, we were hit first. It was beautiful that night because my neighbor had his 12" telescope out in his yard.
However, what is really intriguing is what happened over the next week. Everyday, driving to and from work I would notice all of these FirstEnergy trucks out on the street. No, they were not restoring power. THEY WERE TRIMMING TREES.
The NERC report states that FirstEnergy failed to trim trees to protect the grid. FirstEnergy knew that they had caused the problem and they tried to cover it up.
Also keep in mind that FirstEnergy owns the troubled Davis-Bessie Nuclear plant in Sandusky, Ohio. The nuke plant was shut down almost 2 years ago because boric acid had eaten a football sized hole in the reactor lid. It has been repaired but not re-started due to "safety culture concerns".
Finally, my G/F is a field biologist. She is constantly doing field surveys for FirstEnergy. She always speaks of their poor land use and inability to maintain their lines. She has even seen over-heated lines drooping almost to the ground.
Re: money for u (Score:2)
Let's all wget that link at 2:00 pm EST, and see how the spammer's server handles a Slashdoting.
Re:I can see it now... (Score:2)
Not trolling, but, really do think I heard or read it about it from one of the news sources.
Anyone else know anything like this?
Re:The problem is (AGAIN) government regulation (Score:2)
Though the govt sucks too. Damned if we do, damned if we dont.
Re:The problem is (AGAIN) government regulation (Score:2)
Give it a break will ya...accidents happen. Capitalists stub their toes just as friggin much as socialists do, only when capitalists stub their toes, they blame government for it, and when socialists stub their toes, they blame the lack of government mandates requiring soft fuzzy stripes on the bottom two inches of every wall in the country.
Don't get me wrong, I am definitely much more in favour of less government regulation in all sorts of things, but accidents will still happen, regardless of market fo
Re:The problem is (AGAIN) government regulation (Score:2)
So sever all power ties with the US, since the ontarioans never experience accidents, no
Re:The problem is (AGAIN) government regulation (Score:2)
This isn't a case of stupidity of the gov't. It's probably stupidity of not testing their systems periodically for failover. And not the entire grid testing for failover, which would be ideal, but just one company. Or so says the report
-s
Re:The problem is (AGAIN) government regulation (Score:2)
Then do something about it. Name-calling slashdot readers isn't going to start any sort of political discussion. However, it will make you look like a dumbass.
Re:The problem is (AGAIN) government regulation (Score:2)
What I would want to see here is a country by country and region by region comparison of how power utilities work in various dimensions. Enron for example was a disaster, a corrupt organizations systematically attempting to defraud the public-and Enron as sold originally as a more "free market" solution.
My guess here is that ther
I call bullsh**... (Score:2, Informative)
These emergency lines were never meant for this, and since the companies that were buying and selling power didn't own the lines, they had no motivation to upgrade the capacity.
This [aip.org] article explains alot.
You got it in reverse (Score:5, Insightful)
The deregulated US power grid is overstressed, has little or no margin for extra demand, and has outdated and poorly maintained emergency systems. And thanks to Enron and friends, artificial scarceness has been created to crank up prices.
In a nutshell, you pay more and get less(safety, avaliability, quality) than when the market was regulated. Ain't The Market grand?
Electric power is a basic necessity(in our post-industrial world), and should be managed by the state, just like water works and law enforcement.
Re:You got it in reverse (Score:2, Insightful)
Deregulation can work, but as you seem to imply, only if it is properly thought out. This is a very rare occurance in politics and government.
Re:You got it in reverse (Score:4, Insightful)
Absolutely, and it's interesting to note that planned US spending on electrical infrastructure to 2005 is 71 cents per American per year. Meanwhile, the US gov. plans to spend $255 [guardian.co.uk] per citizen in Iraq! (Health spending plans are also illuminating : $38 per capita on hospitals in Iraq versus $3.30 in the US).
If I was a US citizen, I would be furious about this failure to invest my tax dollars in my own country's infrastructure.
Re:A unix system with X windows (Score:2)
Re:A unix system with X windows (Score:2)
Ok, I guess it could be really really old, but even then, I would have expected any "supported" system to have had X ported to it by the manufacturer. It's not that hard to write the ddx part of an X server (and this mythical old system will almost certainly have a manufacturer-specific framebuffer); and the dix part isn't too hard to compile...
I just think it
Re:X? (Score:2)
Re:Mayor of NYC needs to apologize to Canada (Score:2)
Different people in the US seem to be making a career of pointing fingers at Canada. Terrorists? They came in from Canada! Power outages? Must be the beavers!
I don't care about apologies anymore because they're not sincere, nor does anyone change from them.
Now if the mayor of NYC apologised and did something to improve relations across the border, I might be impressed. In the real world though, all we'll ever get is, "Sorry--it wasn't the Canadians THIS time."
Re:Mayor of NYC needs to apologize to Canada (Score:2)
Subtle difference, Quebecois live in Canada, but do not consider themselves Canadian. Canadians wouldn't do that to children.
Re:Slashdot geeks need to apologize to Microsoft (Score:2)
Anyone that stated that the outage was IN FACT caused by a MS worm SHOULD apologize (as the NYC mayor presented the outage was Canada's fault).
Speculation and suggestion are slightly different... if the mayor simply said "it could be because of Canada" or "at this point it looks like... but we're not sure" things wouldn't be as significant.
Hey, when I accuse somebody of something and then turn out to be wrong, I'll gladly apologize.
MadCow.
Re:X Windows? (Score:2)
'oh no', thought I, not *another* something-run-by-windows fails story.
Imagine my surprise when I read it all. I'm even more surprised it got past the editors.
Re:SCADA systems are DCOM based (Score:2)
Re:SCADA systems are DCOM based (Score:2)
http://www.gepower.com/prod_serv/products/scada_ so ftware/en/downloads/xa21_overview.pdf">
Standards
Standards benefit the vendor as
well as the customer. XA/21? is
built on a foundation of industry
and de-facto standards:
* UNIX(R) * ODBC
* POSIX(R) * DNP 3.0
* UCA? * ICCP
* MVME * SQL
* DXF graphics exchange format
* TCP/IP-based Networking
* ANSI(R) C, FORTRAN
* X Windows Systems
* IEC(R) 870-6 TASE.2
* ELCOM 90
Re:SCADA systems are *NOT ALWAYS* DCOM based (Score:5, Informative)
SCADA systems exist that will run on QNX, Linux (See VSystem [vista-control.com]), Windows NT/2000, DOS (Yes, DOS - i.e. FactoryLink with DesqView) as well as other older platforms like VAX/VMS and PDP-11.
So, SCADA systems do not necessarily rely on OPC or DCOM - some do, some don't. Some still use DDE (trust me, not the most reliable means of communication).
Industry uses all sorts of SCADA systems, from several vendors - Rockwell Software [rockwell.com] makes RSView, Wonderware [wonderware.com] makes InTouch, Siemens [siemens.com] makes WinCC, you get the picture. The GE package involved in the systems at First Energy is only one example of SCADA, in the same way that EMACS is only one example of a text editor.
Sorry for the rant, but IAAAE (I am an automation engineer).