Northrop Grumman Says 'I'm Sorry' For Virginia IT Outage 168
Posted
by
Soulskill
from the wipe-the-tears-away-with-benjamins dept.
from the wipe-the-tears-away-with-benjamins dept.
Lucas123 writes "After a storage area network in a data center run by Northrop Grumman went down last week, crippling 26 state agencies' websites — some for more than a week — Northrop Grumman has now apologized to Virginia, saying it will learn from its mistakes in order to recover systems faster in the future. Northrop's $2.6 billion service contract with Virginia's government has come under harsh criticism in the past for service outages, along with project delays and cost overruns."
My Project (Score:5, Informative)
Re:Poor metropolitan area (Score:1, Informative)
What does the infrastructure of highways have to do with with the State Governments reliance on a single storage area network?
Re:IT Bubble Syndrome (Score:1, Informative)
The problem with DI contractors is that the government rewards good sales teams more than good engineering teams. It means that a disproportionate amount of overhead must be factored in to a winning bid, and since winning bids are often on the lower cost side, it means the engineering teams are always underfunded for the project.
It was EMC storage failure (Score:4, Informative)
It was EMC storage that failed:
http://www.boston.com/business/technology/articles/2010/08/31/emc_system_serving_virginia_breaks_down/
"A major portion of the network shut down on Thursday after some of the EMC gear malfunctioned. As many as 400 server computers in various government departments relied on the storage network and were knocked offline.
Both Northrop Grumman and EMC declined to comment, directing all inquiries about the breakdown to the Virginia Information Technologies Agency, which oversees all of that state’s government computer systems. According to the agency’s website, EMC said that Thursday’s breakdown was unprecedented. “The manufacturer reports that the system and its underlying technology have an exemplary history of reliability, industry-leading data availability of more than 99.999% and no similar failure in one billion hours of run time,’’ the website said."
Re:My Project (Score:4, Informative)
Re:$2.6 billion service contract? (Score:5, Informative)
Here you go, direct from our local news radio station. [wtop.com]
"Northrop Grumman holds a $2.4 billion, 10-year contract with the Virginia Information Technologies Agency to build, operate and maintain the state's 7-year-old, problem-plagued consolidated computer services bureaucracy. It is the largest single-vendor contract in Virginia history. The partnership has been repeatedly criticized in JLARC studies for poor and tardy delivery of services, cost overruns and system failures."
These systems are directly integrated into the DMV, as well as the Department of Social Services and Department of Taxation, amongst others.
Re:My Project (Score:5, Informative)
Unless they have gone to some very lengthy steps to hide this you can probably discover the information on your own.
http://www.dmo.ca/blog/detecting-virtualization-on-linux/ [www.dmo.ca]
This page details steps for many different types of virtualization environments. Though I think it would be just as fast to sort through the output of dmidecode and look for an identification in the mess.
I'm afraid this is rather linux centric, but even so similar data sets can be collected on windows.
Re:$2.6 billion service contract? (Score:5, Informative)
This kind of thing seems to be a growing trend in government IT. I'm posting anonymously because while I don't think I'm going to say anything that violates an NDA, it's better to be vague and sure.
The county that I live in recently made a move like this: they fired basically all of their IT staff and replaced them with the lowest bidding consulting company.
The upside of that from a certain fiscal standpoint is that they've eliminated a bunch of positions with pensions and good (and therefore expensive) benefits. These people have been replaced, in some departments, by the exact same people now subcontracting through the consulting company. This isn't really cheaper -- they won't have additional pension obligations to those people, but they're drawing much higher salaries than before, and obviously the consulting company gets a sizeable cut too.
In other departments, all the long-term employees have been replaced by new consultants. This is a problem in that, probably the people who had those jobs should have documented their networks and tasks much better, but the fact is, they didn't. The whole memory of those departments has been flushed. Inexperienced people are now trying to figure out how to maintain processes that literally no one who works there knows anything about. It's a disaster and it's going to get a lot worse before it gets better.
Meanwhile, the county executive is running for governor, touting the above as a great accomplishment. Hey, he shrunk the size of government by eliminating many permanent positions! By the time people realize that, not always, but sometimes, a lifer IT person is worth their pay because of the institutional memory they have of a thousand important things that were never documented, the election will be over.
Re:NGC Culture (Score:4, Informative)
I do a lot of business with both the federal gov't and private sector businesses on IT projects. You've over-simplified things to the point of painting an inaccurate picture. Federal contracting is extremely complex and there are myriad types of contracts that can be awarded, each with different terms. It sounds like what you're describing is a labor-hour contract. The contractor bills the gov't for the "fully burdened cost" of putting a warm butt in a seat. This includes the worker's salary, overhead, G&A (general & administrative), and profit. All together, it's typically a lot more than a 50% markup of the staff's straight salary.
Unlike most private sector contracts, when doing a fully burdened labor hour contract with the feds, the contractor will spell out exactly what their profit margin is. Generally this is only 6-10%, which is considerably lower than the private sector. Despite what everyone thinks, doing business with the gov't isn't all that lucrative. It's an extremely competitive market in which the bottom-line cost is almost always the most important factor. Contracting officers are even prohibited by law to give preferential treatment to companies that have previously done a great job.
I can't really comment on forced furloughs, because I'm not familiar with how Northrup operates. But just because they do "government contracts" doesn't necessarily mean they can afford to keep highly-skilled staff on the payroll until they find a new project for them. Federal contracts can really help with sales revenue because they can be large awards and the government *always* pays. However, the trade off is all the red tape (which increases G&A costs) and the low profit margins. Next time you hear about Company X getting a $10M contract, don't just roll your eyes. Get a hold of their proposal and the contract and see what their actual profit is on the contract. Both documents are public property and available upon request from the federal contracting officer that made the award. (Defense related contracts might need to be pried from gov't with a FOIA request though)
Re:To be fair... (Score:3, Informative)
Really? This is hardly a great track record:
During the first six months of 2009, Virginia's Department of Transportation (VDOT) experienced 101 significant IT outages totaling 4,677 hours: an average of more than 46 hours per outage. One outage, the Times-Dispatch said, took 360 hours to correct. The state's Department of Motor Vehicles (DMV) has experienced over the course of 5 weeks this autumn some 12 outages that put individual DMV offices out of business for a total of more than 100 hours the paper says.
From here [ieee.org]. I'm sorry, but there is no excuse for that.
Re:It was EMC storage failure (Score:4, Informative)
I would sue the living shit out of Northrop. That's insane.
Based on what would they sue? The contract they signed had no requirement of redundancy [ieee.org]. As much fault as NG has in this, it's not like they broke the contract or anything. This is as much the fault of the incompetents in Virginia's IT Agency as on NG's.
Apparently, when VITA negotiated its 10-year, $2.3 billion outsourcing contract with Northrop Grumman to modernize Virginia's 85 state government agencies' IT systems and networks, it forgot to require network that backup capability be provided in case of network failure, the Richmond Times-Disptach reported over the weekend.
Re:Apt Futurama quote (Score:3, Informative)
If there was "negligence" then there was no cap on the damages. Given what was available initially and what has come out since, there seems to be little doubt to those that matter that the "negligence" standard was met and the $75 million cap would not apply.
And even if it did, that's a cap on punitive, not actual damages. As such, they could be sued for trillions, and setting up an immediate fund for the civil issues has no impact on the small part that the $75 million would cover, if it applies (which it apparently doesn't).
BP did the correct thing and said out front before ANYONE mentioned making them pay - that they would pay for it.
That's why it took a meeting with the President before they actually did anything other than cheap words? Because they did it out of the goodness of their hearts? Or were they asked before they did it? I can't imaging that there'd be a meeting with such an outcome with nothing being mentioned about making them pay. Oh, and "ANYONE" is simply false, as the media was alreaty mentioning it before they announced anything. So you seem to be wrong on every single point you mention. In a rare ad hominem (just calling someone an ass isn't an ad hominem, it's an insult, you have to actually insult them and at least imply that because of the trait insulted that other things they say should be ignored), I'll state that anyone that can't get a single fact right on BP shouldn't be listened to on anything else liability related.
Just because they can get out of paying for something doesn't make it wise to do it.. as for NG.. i doubt the VA contract is a large portion of the biz and isn't a core piece - so yea.. they are going to use cheap words.
They'll use cheap words, but not for the reason you mention. They are a professional government contractor. They know the government rewards incompetence. They'll use this as an excuse to increase rates next time (see, you only gave us 2.4 billion, and it wasn't enough). The professional contractors underperform almost every time. The professional contractors are over budget almost every time. And, as a result, they get paid more for the overruns and get more contracts to fix what wasn't right the first time. But, of course, the problem was predicted quite clearly by Eisenhower. NG is a product of the military industrial complex. But having a problem laid out before us clearly by a president doesn't seem to affect anyone. Washington stated in his farewell address that parties would harm the country, and look where they are taking us now...
Re:Apt Futurama quote (Score:3, Informative)
Nagel is quite clearly correlating the re-cert process with funding the commitments BP as made.
Either he's bluffing (quite possible) or he's making it clear that funding for those programs is contingent on re-cert.
Why? Nagel is clearly insinuating that BP would discontinue payments on those programs if the wells are not re-certified. How is my restatement of his threat overblown? Just because he minces words means the threat is not there?