Northrop Grumman Says 'I'm Sorry' For Virginia IT Outage 168
Posted
by
Soulskill
from the wipe-the-tears-away-with-benjamins dept.
from the wipe-the-tears-away-with-benjamins dept.
Lucas123 writes "After a storage area network in a data center run by Northrop Grumman went down last week, crippling 26 state agencies' websites — some for more than a week — Northrop Grumman has now apologized to Virginia, saying it will learn from its mistakes in order to recover systems faster in the future. Northrop's $2.6 billion service contract with Virginia's government has come under harsh criticism in the past for service outages, along with project delays and cost overruns."
Apt Futurama quote (Score:5, Insightful)
Hermes: What do we do when we break somebody's window?
Dwight: Pay for it?
Hermes: Heavens, no! We apologize! With nice, cheap words.
To be fair... (Score:3, Insightful)
I am not an apologist for Northrop, I am just saying that this is not exactly one-of-a-kind incompetence.
IT Bubble Syndrome (Score:3, Insightful)
Re:To be fair... (Score:4, Insightful)
Meanwhile, the last week has marked the first time where there was really a valid excuse for apparently unmoving lines at Virginia DMV branches... glad I don't have to get my license renewed until 2017. They should be back up by then.
Re:To be fair... (Score:3, Insightful)
I first read the headline as (Score:3, Insightful)
Re:To be fair... (Score:3, Insightful)
Which is why you don't try to implement this broad an IT contract. Was a damnfool idea at the start.
Re:Poor metropolitan area (Score:3, Insightful)
http://www.thesentinel.com/mont/Pepco-Investigation2010-08-12T10-54-42
The governor of MD wrote a letter to Pepco regarding those power outages. As for the snowstorms - its because the DC area historically does not receive snowfall to justify a ROI on snowplows. Instead, they borrow them from the north. Its not like in PA, where if it blizzards overnight, the streets are clear by 6AM so the kids can go to school.
3-year-olds know better than that (Score:3, Insightful)
good thing Northrup Grumman doesn't do anything important, like, say, vital national security support.
oh, wait... .
Re:I was involved with NG on a project once.... (Score:4, Insightful)
Probably but you die on the inside every day you have to work up bullshit excuse why this was bound to happen and why it's a good thing.
Re:My Project (Score:4, Insightful)
As much as Northrup is being bashed here, I don't think this problem is specific to NG, but common to many large contractors and their subs.
In the few times I've worked with subcontractors doing IT for the government I've been unimpressed. Even being one step away from the prime contract seems to allow for many problems, both technical and managerial. Requirements and deadlines aren't met, and they pull the BP-Halliburton-Transocean trick of avoiding responsibility by blaming each other (as well as everyone's favorite scapegoat: the government). Trying to get a subcontractor to build things they way they were supposed to will often require waiting for the next spiral, which means going way over budget.
I understand the difficulty with pushing too hard, punishing contractors who screw up and scaring them away from government work, but it seems we've gone too far in accepting very expensive third-rate work. As much as the public likes to say government can't do anything right, how much worse off would we be than $2.4B in the hole with nothing to show for it but a mediocre datacenter run by amateurs? I don't think I'm asking for much, I haven't touched on the very messy political poisoning of contracting.
More background. (Score:1, Insightful)
can't find my account info sorry.
More background for those interested.
http://www.govtech.com/gt/749378 this is the same state agency which last summer fired the head of the agency because he attempted to make Northrop Grumman comply with its obligations.
For some background on their previous problems try the IEEE.
http://spectrum.ieee.org/riskfactor/computing/it/virginia-information-technologies-agency-believes-in-the-perfect-network-fairy
And yes, this is the same agency which has been help up as an example of the benefits of taking diverse IT services and pulling them into a central cloud.
Re:Poor metropolitan area (Score:2, Insightful)
http://www.thesentinel.com/mont/Pepco-Investigation2010-08-12T10-54-42
The governor of MD wrote a letter to Pepco regarding those power outages. As for the snowstorms - its because the DC area historically does not receive snowfall to justify a ROI on snowplows. Instead, they borrow them from the north. Its not like in PA, where if it blizzards overnight, the streets are clear by 6AM so the kids can go to school.
And the plowing was also poorly prioritized and executed. I live in Alexandria. My residential street was clear to pavement with repeated passes of salt and sand, but Duke Street and Route 1 were an unplowed rutted mess. All because I have a member of the General Assembly living in my neighborhood (that's at least what I've been told by those in the know).
Re:$2.6 billion service contract? (Score:5, Insightful)
Mod parent way up. I've seen it in other places too.
Often when you ask people in the accounting department, they'll say they don't really know what's going on because IT made the system. When you talk to IT, they'll say they don't know because the system was specified by Accounting. The truth is that some smart guys in each department got together and forged some sort of system together. Then the smart guys went on to bigger and better things, while the peons were left with some special voodoo system.
Now you want to move said voodoo system over to a consulting company. The assumption is that the Accountants know every detail of what the old system did. Well, they don't. But nobody is willing to step forward and say that. So the new consultants come along and gosh, nobody knows what the systems do.
Then people ponder why it "doesn't work." Sigh.
This is how shit happens.
Re:Apt Futurama quote (Score:3, Insightful)
Think BP will pay a dime if it sees it can avoid it?
Yes - and they already have. Under law when it happened they are only liable for 75m they have already paid out a hell of a lot more than than that.
Why? because it would cost them more not too. If BP ran with the law at the time and said too bad soo sorry we are only liable for 75m, the media and the politicians would have gone ramped with it in their own name (elect/pay me i'll fight the evil doers BP for you).. instead BP did the correct thing and said out front before ANYONE mentioned making them pay - that they would pay for it. That they would foot the bill, and so far they have.
Just because they can get out of paying for something doesn't make it wise to do it.. as for NG.. i doubt the VA contract is a large portion of the biz and isn't a core piece - so yea.. they are going to use cheap words.
In other news ... (Score:1, Insightful)
NG reports increased profits and record bonuses for key executives
Re:$2.6 billion service contract? (Score:1, Insightful)
Clearly you don't understand how the government works. It's a circle.
Congress gives companies huge contracts -> Companies make mountains of profit -> Companies donate to congressman's compaign -> Congress gives companies huge contracts
Go read Virginia IT's website... (Score:3, Insightful)
Go see it yourself here [virginia.gov].
On Wednesday, August 25, at approximately 3 p.m., the Commonwealth of Virginia experienced an information technology (IT) infrastructure outage that affected 27 of the Commonwealth's 89 agencies and caused 13 percent of the Commonwealth's file servers to fail. The failure was in the equipment used for data storage, commonly known as a storage area network (SAN). Specifically, the SAN that failed was an EMC DMX-3.
According to the manufacturer of the storage system, the events that led to the outage appear to be unprecedented. The manufacturer reports that the system and its underlying technology have an exemplary history of reliability, industry-leading data availability of more than 99.999 percent and no similar failure has occurred in more than one billion hours of run time. A root cause analysis of the failure is currently being conducted.
Anybody else read this like some middle age guy after "finishing a bit too quickly" and telling his , "I swear honey, this the first time this has ever happened to me..."
Apology (Score:3, Insightful)
Re:It was EMC storage failure (Score:3, Insightful)
Actually the lesson is to not let a bunch of incompetents draw up your IT contracts.
Apparently, when VITA negotiated its 10-year, $2.3 billion outsourcing contract with Northrop Grumman to modernize Virginia's 85 state government agencies' IT systems and networks, it forgot to require network that backup capability be provided in case of network failure, the Richmond Times-Disptach reported over the weekend.
I mean really? Requiring redundancy is such a basic requirement that you really have to wonder if the people in VITA even have a brain.
Irony (Score:1, Insightful)
Ironically, Microsoft has a case study touting the Virginia DMV project as a success story.
http://www.microsoft.com/casestudies/Microsoft-SQL-Server-2005-Enterprise-Edition/Virginia-Department-of-Motor-Vehicles/Virginia-DMV-Enhances-Decisions-Boosts-Safety-Through-Integration-with-Other-Agencies/4000004307
Why NGC is so FUBAR (Score:4, Insightful)
Northrop Grumman was cobbled together over several years from about 1999-2006 from Northrop, Grumman, TRW and several other players. It is so dysfunctional because it is composed of so many competing units that don't operate like a single company. In fact, when I briefly worked for them out of college, most of my coworkers were from TRW and hated the idea of being NGC employees.