Forgot your password?
typodupeerror
Networking IT

Northrop Grumman Says 'I'm Sorry' For Virginia IT Outage 168

Posted by Soulskill
from the wipe-the-tears-away-with-benjamins dept.
Lucas123 writes "After a storage area network in a data center run by Northrop Grumman went down last week, crippling 26 state agencies' websites — some for more than a week — Northrop Grumman has now apologized to Virginia, saying it will learn from its mistakes in order to recover systems faster in the future. Northrop's $2.6 billion service contract with Virginia's government has come under harsh criticism in the past for service outages, along with project delays and cost overruns."
This discussion has been archived. No new comments can be posted.

Northrop Grumman Says 'I'm Sorry' For Virginia IT Outage

Comments Filter:
  • Apt Futurama quote (Score:5, Insightful)

    by dkleinsc (563838) on Friday September 03, 2010 @11:24AM (#33465052) Homepage

    Hermes: What do we do when we break somebody's window?
    Dwight: Pay for it?
    Hermes: Heavens, no! We apologize! With nice, cheap words.

  • by Pojut (1027544) on Friday September 03, 2010 @11:26AM (#33465094) Homepage

    The Maryland/VA/DC metro area is really starting to go down hill, from an infrastructure standpoint. Things are just falling apart around here...oh, and what's that? Instead of investing in fixing aging infrastructure, they instead are spending billions to build the ICC? [iccproject.com] Oh, and what's that? It's STILL going to be a toll road?

    I've lived in Montgomery County my whole life, but I'm quickly getting tired of this place -_-;;

    • by Shimmer (3036)

      Another Montgomery County resident here, and I second this view. We've had an insane number of power outages, water main breaks, crippling snowstorms with unplowed roads, etc. in the last few years.

      I love where I live, though. Who needs electricity?

      • Re: (Score:3, Insightful)

        by Wiarumas (919682)

        http://www.thesentinel.com/mont/Pepco-Investigation2010-08-12T10-54-42

        The governor of MD wrote a letter to Pepco regarding those power outages. As for the snowstorms - its because the DC area historically does not receive snowfall to justify a ROI on snowplows. Instead, they borrow them from the north. Its not like in PA, where if it blizzards overnight, the streets are clear by 6AM so the kids can go to school.

        • by Shimmer (3036)

          Yes, and the power outages are caused by overhead (rather than underground) power lines that are severed when one of our many large, beautiful trees fall on them. However, I think that if you compare this area to others with similar climate/foliage, we still fall short. The large state roads around here were not plowed for DAYS following the Snowpocalypse this past winter. (And then when they were finally plowed, the plowing was done poorly and left many roads with huge ruts and sudden narrowings.) I've liv

        • Re: (Score:2, Insightful)

          by Anonymous Coward

          http://www.thesentinel.com/mont/Pepco-Investigation2010-08-12T10-54-42

          The governor of MD wrote a letter to Pepco regarding those power outages. As for the snowstorms - its because the DC area historically does not receive snowfall to justify a ROI on snowplows. Instead, they borrow them from the north. Its not like in PA, where if it blizzards overnight, the streets are clear by 6AM so the kids can go to school.

          And the plowing was also poorly prioritized and executed. I live in Alexandria. My residential street was clear to pavement with repeated passes of salt and sand, but Duke Street and Route 1 were an unplowed rutted mess. All because I have a member of the General Assembly living in my neighborhood (that's at least what I've been told by those in the know).

        • Yeah, here in New York (state) we kinda laugh at the mid-south states those close over a couple inches. granted, her, we seem to be too slow to close stuff.

    • "Northrop's $2.6 billion service contract with Virginia's government..."

      What could they possibly be doing for Virginia that should cost $2.6 billion?
      • by Pojut (1027544) on Friday September 03, 2010 @11:47AM (#33465376) Homepage

        Here you go, direct from our local news radio station. [wtop.com]

        "Northrop Grumman holds a $2.4 billion, 10-year contract with the Virginia Information Technologies Agency to build, operate and maintain the state's 7-year-old, problem-plagued consolidated computer services bureaucracy. It is the largest single-vendor contract in Virginia history. The partnership has been repeatedly criticized in JLARC studies for poor and tardy delivery of services, cost overruns and system failures."

        These systems are directly integrated into the DMV, as well as the Department of Social Services and Department of Taxation, amongst others.

        • Still, what could they possibly be doing for that amount of money. If you work it out, that's almost 30 thousand dollars every hour for 10 years (I didn't even take out weekends or holidays or anything). That's ~890,000 per month. I could run one bangin' IT organization on 240 million dollars a year. Hell, the company I work for now (which is in the top 10 on the Fortune 500 list), our yearly IT budget is smaller than that.
          • Still, what could they possibly be doing for that amount of money.

            Lining their pockets. What else?

          • by Pojut (1027544)

            Didn't say VA was getting their money's worth, just wanted to tell people what they were doing :-)

          • Still, what could they possibly be doing for that amount of money. If you work it out, that's almost 30 thousand dollars every hour for 10 years (I didn't even take out weekends or holidays or anything). That's ~890,000 per month.

            That's about $20 million/month ($240 million per year / 12 months/year).

      • by Anonymous Coward on Friday September 03, 2010 @12:35PM (#33466028)

        This kind of thing seems to be a growing trend in government IT. I'm posting anonymously because while I don't think I'm going to say anything that violates an NDA, it's better to be vague and sure.

        The county that I live in recently made a move like this: they fired basically all of their IT staff and replaced them with the lowest bidding consulting company.

        The upside of that from a certain fiscal standpoint is that they've eliminated a bunch of positions with pensions and good (and therefore expensive) benefits. These people have been replaced, in some departments, by the exact same people now subcontracting through the consulting company. This isn't really cheaper -- they won't have additional pension obligations to those people, but they're drawing much higher salaries than before, and obviously the consulting company gets a sizeable cut too.

        In other departments, all the long-term employees have been replaced by new consultants. This is a problem in that, probably the people who had those jobs should have documented their networks and tasks much better, but the fact is, they didn't. The whole memory of those departments has been flushed. Inexperienced people are now trying to figure out how to maintain processes that literally no one who works there knows anything about. It's a disaster and it's going to get a lot worse before it gets better.

        Meanwhile, the county executive is running for governor, touting the above as a great accomplishment. Hey, he shrunk the size of government by eliminating many permanent positions! By the time people realize that, not always, but sometimes, a lifer IT person is worth their pay because of the institutional memory they have of a thousand important things that were never documented, the election will be over.

        • by AB3A (192265) on Friday September 03, 2010 @12:54PM (#33466298) Homepage Journal

          Mod parent way up. I've seen it in other places too.

          Often when you ask people in the accounting department, they'll say they don't really know what's going on because IT made the system. When you talk to IT, they'll say they don't know because the system was specified by Accounting. The truth is that some smart guys in each department got together and forged some sort of system together. Then the smart guys went on to bigger and better things, while the peons were left with some special voodoo system.

          Now you want to move said voodoo system over to a consulting company. The assumption is that the Accountants know every detail of what the old system did. Well, they don't. But nobody is willing to step forward and say that. So the new consultants come along and gosh, nobody knows what the systems do.

          Then people ponder why it "doesn't work." Sigh.

          This is how shit happens.

          • by hey! (33014)

            If accounting doesn't know what's going on, then they're incompetent. Accounting is essentially financial epistemology: it is about creating true, *justifiable* beliefs. The ultimate test is the audit. An *outsider* comes in and takes everything you say about yourself financially and puts it to the test. Then they sample things that happened (e.g. an invoice payment) and make every event is reflected on the financial statements.

            When you have a situation where ignorance is tolerated in the accounting depa

            • by AB3A (192265)

              Read my sig. It explains everything.

              Yes, they're incompetent. If they were really good they wouldn't stick around. Most places have a difficult time justifying the sorts of salaries that keep the good people on staff.

              Look, the problem is that large organizations have often bought in to the notion that Human Resources knows what it's doing. They do not. They can not. I've watched this process in my own company. It's not pretty. It is a process designed by lawyer/MBA dweebs where success is almost always acci

    • by astar (203020)

      Maybe, but you may be parochial.

      I just heard today about a town who put the entire fire department on half-time and at minimum wage. Infrastructure has been a know problem for decades and more recently even a matter of considerable public concern, as when an interstate bridge collapsed a few years back. But big cuts in public safety are new. And there is a nice pattern. The cuts are not something that make the mass media. but I saw a long summary compiled from local media last month, so I figure this is

      • by bsDaemon (87307)

        Where are you going to find capable individuals willing to run into burning building to save strangers on half time for minimum wage? That's the most asinine thing I've ever heard. better to be a volunteer than a sucker.

        • by astar (203020)

          you are of course right. But be a little more realistic. With talking heads saying real unemployment is 22% and a better number being 30%, then consider all the really good reasons you can up with for cutting police resources :-) I am old enough to remember lots of federal troops and tanks in the street. And back in last crash, I believe MacAuthur was shooting people in the capital mall. So, before my time, but I googled a bit with "macauthur general riot suppression". I did not quite see "shooting".

          S

    • I'd be a hell of a lot more concerned with your state's taxation, police run amok, and unionization of the public servants. MD is a joke.

  • Pretty weak apology if you ask me. I guess these military contractors are used to the "boys will be boys" pat on the hand they get from the Department of Defense when they screw up there.

    • Some (many?) of DoD projects are "high-risk" of failure. They are typically want to push the envelope and be early adopters of technology before their enemy does.

      I don't know how they run their IS/IM side of the house (payroll, HR, administrative IT).

      Plus NGC and co know DoD has very few obligations to make their budget (per project) overruns public, with the easily cited "national security" concerns.

  • My Project (Score:5, Informative)

    by WED Fan (911325) <akahige&trashmail,net> on Friday September 03, 2010 @11:28AM (#33465124) Homepage Journal
    I have a project in a separate NG hosted dataspace, in Virginia. They are killing us with incompetence and their sub contractors are worse then they are. We are still trying to get things certified and they won't provide us information about their hosting. We think they have us on virtual servers that belong to another project, and the reason they don't want to tell us anything is that it would reveal they are in breach, since we are paying for dedicated servers.
    • Re:My Project (Score:4, Informative)

      by snspdaarf (1314399) on Friday September 03, 2010 @11:44AM (#33465332)
      I do some work with NG sub contractors. One is a delight to work with. Working with the other is like staring at the sun through binoculars. People don't answer email, sometimes because they are lazy, sometimes because they quit. When an answer does come in, it is worse than none at all. It's like the worst troll postings on slashdot, full of errors, run-on sentences that confuse more than clarify, and off topic. Absolute nightmare.
      • Re:My Project (Score:4, Insightful)

        by Monchanger (637670) on Friday September 03, 2010 @12:29PM (#33465930) Journal

        As much as Northrup is being bashed here, I don't think this problem is specific to NG, but common to many large contractors and their subs.

        In the few times I've worked with subcontractors doing IT for the government I've been unimpressed. Even being one step away from the prime contract seems to allow for many problems, both technical and managerial. Requirements and deadlines aren't met, and they pull the BP-Halliburton-Transocean trick of avoiding responsibility by blaming each other (as well as everyone's favorite scapegoat: the government). Trying to get a subcontractor to build things they way they were supposed to will often require waiting for the next spiral, which means going way over budget.

        I understand the difficulty with pushing too hard, punishing contractors who screw up and scaring them away from government work, but it seems we've gone too far in accepting very expensive third-rate work. As much as the public likes to say government can't do anything right, how much worse off would we be than $2.4B in the hole with nothing to show for it but a mediocre datacenter run by amateurs? I don't think I'm asking for much, I haven't touched on the very messy political poisoning of contracting.

        • Re: (Score:2, Interesting)

          by Anonymous Coward

          I *was* a NG subcontractor, doing sysadmin work in a datacenter in a different state. Oh, the stories I could tell. More than once, as a new project was in the bidding stage, I would point out serious, game-stopping flaws in the proposal. The response was inevitably along the lines of "Fixing that would require us to raise our bid, and we don't have any exposure this way because we can blame the agency for a flawed proposal. Then they would win the bid, a bunch of equipment would get dumped on the floor, th

          • Getting charged out at $100/hour while getting paid $10-14/hour was depressing, especially since the contracting company did nothing in that case apart from collect and hand out money (insurance, health care etc was my responsibility). When the client (actually former workplace encumbered by a requirement for layoffs and a hiring freeze) looked as if they would be able to employ me directly the contract company took near criminal steps to poison my relationship with that company. I left after being blamed
      • Re: (Score:3, Interesting)

        by guruevi (827432)

        A couple of months ago there was a declassified document (either on /. or Wikileaks) that was basically a CIA/War Department handbook for (civilian) saboteurs against an oppressive regime. If you read this, you can see a lot of the practices that are recommended to be used against the 'enemy' are exactly the same as those contractors use. The goal is to have the enemy (or the company you're contracted to) spend as much money as possible on lost time and resources. The only difference is the people/group tha

    • Re:My Project (Score:5, Informative)

      by Cylix (55374) * on Friday September 03, 2010 @11:54AM (#33465452) Homepage Journal

      Unless they have gone to some very lengthy steps to hide this you can probably discover the information on your own.

      http://www.dmo.ca/blog/detecting-virtualization-on-linux/ [www.dmo.ca]

      This page details steps for many different types of virtualization environments. Though I think it would be just as fast to sort through the output of dmidecode and look for an identification in the mess.

      I'm afraid this is rather linux centric, but even so similar data sets can be collected on windows.

      • by WED Fan (911325)
        It will tell we are virtualized, but it won't tell us if they are putting other systems on our dedicated systems, or if we are on someone elses.
        • by h4rr4r (612664)

          Try to use up all the IO, bonnie++ is good for that, to what you assume are local discs.

          • by Cylix (55374) *

            Hammer it!

            Bonnie is good for a straight disk measurement, but depending on the options set it may not be too useful. Depending on the configuration you may be able to burst large quantities over any thresholds.

            Really, a few baselines need to be taken and compared against the cluster.

            I would perform a few isolated IOZone and Orion benchmarks to set the baseline. Once this completed a parallel initiated test of all hosts will determine which units may be shared.

            Now, if all the hardware is the same it should l

  • To be fair... (Score:3, Insightful)

    by betterunixthanunix (980855) on Friday September 03, 2010 @11:29AM (#33465144)
    To be fair, there is no evidence that Northrop is doing worse than anyone else would have done. We are talking about an enormously complex IT infrastructure here (or so I assume, since it is a government network), and this is not exactly a uniquely bad failure. A week may seem extreme, but I have seen smaller scale systems go down for that long.

    I am not an apologist for Northrop, I am just saying that this is not exactly one-of-a-kind incompetence.
    • Re:To be fair... (Score:4, Insightful)

      by bsDaemon (87307) on Friday September 03, 2010 @11:32AM (#33465170)

      Meanwhile, the last week has marked the first time where there was really a valid excuse for apparently unmoving lines at Virginia DMV branches... glad I don't have to get my license renewed until 2017. They should be back up by then.

    • Re: (Score:3, Insightful)

      by rotide (1015173)
      Frankly, I've been witness to whole Enterprise datacenters going offline and then being brought up system by system in less than a week (candy red button). A week to fix a SAN issue? really? Why not classify it as a Sev0 (public exposure) (probably a somewhat unique code to the company I work for) and get the vendors in THAT DAY to fix it?
      • Frankly, I've been witness to whole Enterprise datacenters going offline and then being brought up system by system in less than a week (candy red button). A week to fix a SAN issue? really? Why not classify it as a Sev0 (public exposure) (probably a somewhat unique code to the company I work for) and get the vendors in THAT DAY to fix it?

        Depends on how much storage was in the SAN and the nature of the failure. If it was as easy as replacing a memory card then your point it completely valid. If, however, there was some data corruption then there might have been terabytes of data that needed to be verified and recovered. The article states that 26 state agencies were affected, "some for more than a week" which implies a variable amount of time. Without having more detail, it could have been that some were back up and running the same day

    • by Cylix (55374) *

      I think the problem is there is more then one complain against the company.

      Still, 2.6 billion I could build way past four nines of availability.

      I currently maintain better availability on a budget that is way smaller and the equipment would not be considered inexpensive. In fact, a failure on that scale would blow any availability numbers I have out of the water for years. Short of data corruption that works its way into the system a single SAN failure should not halt operations for a week.

      Now, instead of

    • Re: (Score:3, Insightful)

      by Spazmania (174582)

      Which is why you don't try to implement this broad an IT contract. Was a damnfool idea at the start.

    • Re: (Score:3, Informative)

      Really? This is hardly a great track record:

      During the first six months of 2009, Virginia's Department of Transportation (VDOT) experienced 101 significant IT outages totaling 4,677 hours: an average of more than 46 hours per outage. One outage, the Times-Dispatch said, took 360 hours to correct. The state's Department of Motor Vehicles (DMV) has experienced over the course of 5 weeks this autumn some 12 outages that put individual DMV offices out of business for a total of more than 100 hours the paper says.

      From here [ieee.org]. I'm sorry, but there is no excuse for that.

      • by sjames (1099)

        The sad thing is that there are many thousands of mom'n'pop IT shops that do much better than that on a shoestring budget (when scaled proportionally) that would really love to get even part of a contract like that but never will because they always go to the behemoths filled with fail like NG.

    • To be fair, there is no evidence that Northrop is doing worse than anyone else would have done.

      They are doing significantly worse than the state's internal people before NG took over. And at a much higher price. There were lotsof other inexcusable outages and failures before this one.

      The worst part is that even knowing what a terrible deal VA was getting from NG, the governor decided to extend the contract for another 3 years and allow NG to increase their fees. If these guys had been working for a private company, they would have been thrown out long ago, and any executive suggesting they should

      • But...but...smaller government!!! Private industry is always better and more efficient than the government. That is, except for the all cases like this where they really aren't.

  • IT Bubble Syndrome (Score:3, Insightful)

    by darien.train (1752510) on Friday September 03, 2010 @11:31AM (#33465154) Journal
    The "old business man discovering the internet" IT bubble culture is still alive and well in the defense industry. They have such a bad track record with networking technology it borders on scary. Transformation [pbs.org] comes to mind quickly and they keep repeating the same mistakes.
    • by X0563511 (793323)

      I was wondering what business an aerospace company has doing providing IT services for others.

      I can understand diversity... but this is completely out of the aerospace "tree" !!

  • by quangdog (1002624) <quangdog@@@gmail...com> on Friday September 03, 2010 @11:36AM (#33465226)
    Fresh out of school with my CS degree I went to work on a project for my employer that involved partnering and working directly with folks from NG. The original deadline for shipping the solution was something like 6 months after I started. The complete and utter incompetence of the NG side of things wound up stretching this out more than 18 MONTHS longer, and the final delivery lacked a lot of the original stated requirements.

    Being the newbie to the whole corporate culture, I was shocked that people were not bothered at all by blown deadlines, missed estimates, and huge cost overruns. Shortly after the project finally delivered, I bailed to work for a much smaller company (fewer than 10 employees) where I discovered that I really love the smaller, more dynamic environment that only small companies can provide.

    Working for huge corporations just sucks.
  • NGC Culture (Score:5, Interesting)

    by CherniyVolk (513591) on Friday September 03, 2010 @11:40AM (#33465292)

    I work with a fair number of ngc.com; I'm a contractor myself. At one point, I had a few interviews for Northrop and I'm so glad they over looked my talents because they seem to abuse the talent they have.

    Now, this may not be for every department or division, but almost every NGC employee I know is basically well familiar with furlough. Whether good or bad, NGC is left with the ability to place entire departments on furlough to reduce overhead costs in the event a contract dries up. Now perhaps it's their size, perhaps they simply don't care about their workers, but this sort of thing seems to happen often. I'd guess that no NGC employee with a tenure more than 2 years hasn't been out of work for up to a month or so. But this is how things are run there.

    See, government contracting works like this. You create a company, hire some folk to work on a contract. Whatever their salary is, you charge the government +50% or more, so essentially the government is not only flat out paying your salary but also the company for your services. If the contract ends, so does your job as the company may not want to charge overhead. In contrast to other business sectors, employment typically isn't grounded so harshly on the existence of a contract, which is where cost of business and business management can keep workers afloat even during down times (think department store).

    I only point out Northrop because while all government contracting is essentially this contract+play model, Northrop has a reputation of placing people on furlough much more often than other companies such as CACI, Raytheon, General Atomics etc. Some Northrop employees seem to live the lives similar to actors and actresses in Hollywood, and I'm not talking about Tom Hanks acting, but maybe those actors that get little spots from time to time on your sitcoms. They literally live in apartments, and wait for the phone to ring day after day. Northrop employees seem to wake up in the morning, wondering if they'll still have a job at the end of the day.

    What does this observation have to do with the op? Well, it seems that moral and motivation might be a bit low on a large scale at Northrop, so such blunders are no surprise to me.

    • by Spazmania (174582)

      NGC is what we in the business refer to as a "body shop."

      http://www.realrates.com/bbs/messages/tips4.htm [realrates.com]

    • Re:NGC Culture (Score:4, Informative)

      by XorNand (517466) on Friday September 03, 2010 @12:36PM (#33466052)

      See, government contracting works like this. You create a company, hire some folk to work on a contract. Whatever their salary is, you charge the government +50% or more, so essentially the government is not only flat out paying your salary but also the company for your services. If the contract ends, so does your job as the company may not want to charge overhead. In contrast to other business sectors, employment typically isn't grounded so harshly on the existence of a contract, which is where cost of business and business management can keep workers afloat even during down times (think department store).

      I do a lot of business with both the federal gov't and private sector businesses on IT projects. You've over-simplified things to the point of painting an inaccurate picture. Federal contracting is extremely complex and there are myriad types of contracts that can be awarded, each with different terms. It sounds like what you're describing is a labor-hour contract. The contractor bills the gov't for the "fully burdened cost" of putting a warm butt in a seat. This includes the worker's salary, overhead, G&A (general & administrative), and profit. All together, it's typically a lot more than a 50% markup of the staff's straight salary.

      Unlike most private sector contracts, when doing a fully burdened labor hour contract with the feds, the contractor will spell out exactly what their profit margin is. Generally this is only 6-10%, which is considerably lower than the private sector. Despite what everyone thinks, doing business with the gov't isn't all that lucrative. It's an extremely competitive market in which the bottom-line cost is almost always the most important factor. Contracting officers are even prohibited by law to give preferential treatment to companies that have previously done a great job.

      I can't really comment on forced furloughs, because I'm not familiar with how Northrup operates. But just because they do "government contracts" doesn't necessarily mean they can afford to keep highly-skilled staff on the payroll until they find a new project for them. Federal contracts can really help with sales revenue because they can be large awards and the government *always* pays. However, the trade off is all the red tape (which increases G&A costs) and the low profit margins. Next time you hear about Company X getting a $10M contract, don't just roll your eyes. Get a hold of their proposal and the contract and see what their actual profit is on the contract. Both documents are public property and available upon request from the federal contracting officer that made the award. (Defense related contracts might need to be pried from gov't with a FOIA request though)

      • by Rich0 (548339)

        I think it depends on what you consider "profit."

        If you consider the money going to the shareholders, then sure, you're right.

        However, government contractors aren't run to benefit the shareholders - they're run to benefit the executives. By building empires of lower employees the executives justify their own positions, their large salaries, and they get armies of people they can rule over.

        I'd consider any cost beyond the basic time and materials, and minimal overhead for basic facilities, to be "profit" in

      • I do a lot of business with both the federal gov't and private sector businesses on IT projects. You've over-simplified things to the point of painting an inaccurate picture.

        You are correct, the contracts can be very complex. There are all kinds of things that govern contracts, and I didn't mean to belittle this. I just wanted to over-simplify the general structure as it's here that the cause and effect of furloughs, layoffs and such persist.

        I'm also somewhat aware of contract awards, primarily from the m

    • by wkcole (644783)

      Now, this may not be for every department or division, but almost every NGC employee I know is basically well familiar with furlough. Whether good or bad, NGC is left with the ability to place entire departments on furlough to reduce overhead costs in the event a contract dries up. Now perhaps it's their size, perhaps they simply don't care about their workers, but this sort of thing seems to happen often. I'd guess that no NGC employee with a tenure more than 2 years hasn't been out of work for up to a month or so. But this is how things are run there.

      See, government contracting works like this. You create a company, hire some folk to work on a contract. Whatever their salary is, you charge the government +50% or more, so essentially the government is not only flat out paying your salary but also the company for your services. If the contract ends, so does your job as the company may not want to charge overhead. In contrast to other business sectors, employment typically isn't grounded so harshly on the existence of a contract, which is where cost of business and business management can keep workers afloat even during down times (think department store).

      FWIW, this makes it sound like NGC is a couple of steps better than the contract IT shops that I'm familiar with that service the private sector. 100% markup is fairly common, and keeping idle employees "on furlough" is a concept that doesn't really exist. When the contract ends, the job ends. A pure pimp agency will usually try to place a profitable contractor in another spot ASAP once a customer provides notice, but they won't guarantee anyones rates and I have never heard of a pimp agency doing anythin

    • Re: (Score:2, Interesting)

      by Anonymous Coward

      Whoa anecdote! I've worked for DoD contractors for years (and on different coasts). I've never heard of anyone at Northrop being placed on furlogh so I guess we don't know any of the same employees. In my experience, when there are no contracts, the employer covers your salary until the numbers tell them to start laying people off. If you are an employee they either pay you or lay you off. You don't sit around at home waiting for a call. Thats one of the benefits of a big company. They have enough stuff go

  • by Anonymous Coward on Friday September 03, 2010 @11:42AM (#33465318)

    It was EMC storage that failed:

    http://www.boston.com/business/technology/articles/2010/08/31/emc_system_serving_virginia_breaks_down/

    "A major portion of the network shut down on Thursday after some of the EMC gear malfunctioned. As many as 400 server computers in various government departments relied on the storage network and were knocked offline.

    Both Northrop Grumman and EMC declined to comment, directing all inquiries about the breakdown to the Virginia Information Technologies Agency, which oversees all of that state’s government computer systems. According to the agency’s website, EMC said that Thursday’s breakdown was unprecedented. “The manufacturer reports that the system and its underlying technology have an exemplary history of reliability, industry-leading data availability of more than 99.999% and no similar failure in one billion hours of run time,’’ the website said."

    • by TheLink (130905)
      > Both Northrop Grumman and EMC declined to comment,
      > The manufacturer reports that the system and its underlying technology have an exemplary history of reliability

      So who's lying ;).
    • by AB3A (192265)

      Lesson: Don't put all your eggs in one basket. Nothing new about that...

      • Re: (Score:3, Insightful)

        Actually the lesson is to not let a bunch of incompetents draw up your IT contracts.

        Apparently, when VITA negotiated its 10-year, $2.3 billion outsourcing contract with Northrop Grumman to modernize Virginia's 85 state government agencies' IT systems and networks, it forgot to require network that backup capability be provided in case of network failure, the Richmond Times-Disptach reported over the weekend.

        I mean really? Requiring redundancy is such a basic requirement that you really have to wonder if the people in VITA even have a brain.

        • Re: (Score:3, Interesting)

          by dbIII (701233)
          It's such a basic requirement that you would think you would get it for that price. It sounds like using a loophole after being caught out cutting corners to me.
    • by realmolo (574068)

      So, for their $2.6 billion, Virginia doesn't get redundant systems? Everything is on ONE SAN?!

      I would sue the living shit out of Northrop. That's insane.

      • by Lunix Nutcase (1092239) on Friday September 03, 2010 @01:16PM (#33466554)

        I would sue the living shit out of Northrop. That's insane.

        Based on what would they sue? The contract they signed had no requirement of redundancy [ieee.org]. As much fault as NG has in this, it's not like they broke the contract or anything. This is as much the fault of the incompetents in Virginia's IT Agency as on NG's.

        Apparently, when VITA negotiated its 10-year, $2.3 billion outsourcing contract with Northrop Grumman to modernize Virginia's 85 state government agencies' IT systems and networks, it forgot to require network that backup capability be provided in case of network failure, the Richmond Times-Disptach reported over the weekend.

        • Also, for those of you who don't work based off government contracts: If you can't justify a project as coming under something defined in the contract, you don't do it. It's not about laziness, incompetence, or greed, it's because the government is full of rules lawyers that will start to jump up and down and demand to know why you spent $X on something not in the contract. Working for $Unnamed_government_contractor, I will say that one thing they make very clear is that you do not charge time from one proj
      • Referring to an EMC system as "one SAN" is (probably) a gross oversimplification. EMC are the storage vendor you go to when money is no object, and you cannot, absolutely cannot have downtime. If Northrop had the system configured correctly, this is a huge black mark against EMC.

    • by Klinky (636952)

      How are they counting one billion hours of run time? Are they counting uptime and multiplying it by each disk or something? One billion hours is 114,079 years... Five-nines for 1 billions hours is 1.14 years... At that rate they should be happy it only took Northrup a week to get things working again.

    • where I come from, you swap out the suspicious or dead junk, and roll the data back on from backups.

      this, of course, requires you know how to find dead junk, swap it, and retrieve tapes that you knew how to make, and roll them back on according to your training.

      remove all of the above, and it sounds like the error they have.

      monkeys cost less, and can do exactly the same job. just sayin'....

    • by bored (40072)

      There were some other details about how it was a "memory board" that caused the problem an how the fail-over system didn't work properly. So on paper no one can be blamed.

      To me this now seems SOP in the computer industry. Consolidation is king without regard for the idea that consolidation tends to form giant single points of failure. Furthermore, backups and redundant systems are rarely tested. Even when properly tested, fail-over is pure luck as the actual failure case may not be as clear cut as the fail-

  • by edwebdev (1304531) on Friday September 03, 2010 @11:49AM (#33465414)
    "Northrop Grumman Says 'I'm Sorry' For Virginia IT Outrage"
  • This is happening because all of the best IT talent these days are doing startups or working at cool companies like Google and Facebook. This means that Northrup Grumann is staffing their teams with people that can't get those cooler IT jobs. That's the real cause of this disaster for Virginia.
  • by swschrad (312009) on Friday September 03, 2010 @12:20PM (#33465782) Homepage Journal

    good thing Northrup Grumman doesn't do anything important, like, say, vital national security support.

    oh, wait... .

    • I wonder how much that's actually related. When I read the contractor was NG, I asked myself "what the hell are they in that business for?"

      The answer, money, is of course, a major suspect in the failure.

      This contract should not have gone to someone like NG, who probably just turned around and sold it to EMC who should have been competent. That is, if it really was EMC and not one of their subcontractors.

      • But according [northropgrumman.com] to Northrup Grumman they were "recently recognized by the National Association of Counties for Outstanding Achievement in the area of "Information Technology in State Government – Enterprise IT Management Initiatives".

        • Ha! Good one! Ask Northrop what they think of themselves. Mod that +2 Funny.

          That quote cuts off an important bit: "In fact, our partnership with the Virginia Information Technologies Agency was recently recognized" (emphasis mine). So if that's this same project it's not like we're talking about that recognition as a proven track record, though other examples may exist. And this is, in fact, the same project, as you'll can see from the figures ($1.9B over 10 years, which apparently grew to $2.4B) show

  • by HockeyPuck (141947) on Friday September 03, 2010 @01:06PM (#33466446)

    Go see it yourself here [virginia.gov].

    On Wednesday, August 25, at approximately 3 p.m., the Commonwealth of Virginia experienced an information technology (IT) infrastructure outage that affected 27 of the Commonwealth's 89 agencies and caused 13 percent of the Commonwealth's file servers to fail. The failure was in the equipment used for data storage, commonly known as a storage area network (SAN). Specifically, the SAN that failed was an EMC DMX-3.

    According to the manufacturer of the storage system, the events that led to the outage appear to be unprecedented. The manufacturer reports that the system and its underlying technology have an exemplary history of reliability, industry-leading data availability of more than 99.999 percent and no similar failure has occurred in more than one billion hours of run time. A root cause analysis of the failure is currently being conducted.

    Anybody else read this like some middle age guy after "finishing a bit too quickly" and telling his , "I swear honey, this the first time this has ever happened to me..."

  • Apology (Score:3, Insightful)

    by Curunir_wolf (588405) on Friday September 03, 2010 @01:15PM (#33466536) Homepage Journal
    So Virginia taxpayers will continue to get screwed, but Northrop Grumman has now extended a reach-around?
  • by MikeRT (947531) on Friday September 03, 2010 @02:14PM (#33467374) Homepage

    Northrop Grumman was cobbled together over several years from about 1999-2006 from Northrop, Grumman, TRW and several other players. It is so dysfunctional because it is composed of so many competing units that don't operate like a single company. In fact, when I briefly worked for them out of college, most of my coworkers were from TRW and hated the idea of being NGC employees.

A CONS is an object which cares. -- Bernie Greenberg.

Working...