Northrop Grumman Says 'I'm Sorry' For Virginia IT Outage 168
Posted
by
Soulskill
from the wipe-the-tears-away-with-benjamins dept.
from the wipe-the-tears-away-with-benjamins dept.
Lucas123 writes "After a storage area network in a data center run by Northrop Grumman went down last week, crippling 26 state agencies' websites — some for more than a week — Northrop Grumman has now apologized to Virginia, saying it will learn from its mistakes in order to recover systems faster in the future. Northrop's $2.6 billion service contract with Virginia's government has come under harsh criticism in the past for service outages, along with project delays and cost overruns."
Poor metropolitan area (Score:3, Interesting)
The Maryland/VA/DC metro area is really starting to go down hill, from an infrastructure standpoint. Things are just falling apart around here...oh, and what's that? Instead of investing in fixing aging infrastructure, they instead are spending billions to build the ICC? [iccproject.com] Oh, and what's that? It's STILL going to be a toll road?
I've lived in Montgomery County my whole life, but I'm quickly getting tired of this place -_-;;
I was involved with NG on a project once.... (Score:5, Interesting)
Being the newbie to the whole corporate culture, I was shocked that people were not bothered at all by blown deadlines, missed estimates, and huge cost overruns. Shortly after the project finally delivered, I bailed to work for a much smaller company (fewer than 10 employees) where I discovered that I really love the smaller, more dynamic environment that only small companies can provide.
Working for huge corporations just sucks.
NGC Culture (Score:5, Interesting)
I work with a fair number of ngc.com; I'm a contractor myself. At one point, I had a few interviews for Northrop and I'm so glad they over looked my talents because they seem to abuse the talent they have.
Now, this may not be for every department or division, but almost every NGC employee I know is basically well familiar with furlough. Whether good or bad, NGC is left with the ability to place entire departments on furlough to reduce overhead costs in the event a contract dries up. Now perhaps it's their size, perhaps they simply don't care about their workers, but this sort of thing seems to happen often. I'd guess that no NGC employee with a tenure more than 2 years hasn't been out of work for up to a month or so. But this is how things are run there.
See, government contracting works like this. You create a company, hire some folk to work on a contract. Whatever their salary is, you charge the government +50% or more, so essentially the government is not only flat out paying your salary but also the company for your services. If the contract ends, so does your job as the company may not want to charge overhead. In contrast to other business sectors, employment typically isn't grounded so harshly on the existence of a contract, which is where cost of business and business management can keep workers afloat even during down times (think department store).
I only point out Northrop because while all government contracting is essentially this contract+play model, Northrop has a reputation of placing people on furlough much more often than other companies such as CACI, Raytheon, General Atomics etc. Some Northrop employees seem to live the lives similar to actors and actresses in Hollywood, and I'm not talking about Tom Hanks acting, but maybe those actors that get little spots from time to time on your sitcoms. They literally live in apartments, and wait for the phone to ring day after day. Northrop employees seem to wake up in the morning, wondering if they'll still have a job at the end of the day.
What does this observation have to do with the op? Well, it seems that moral and motivation might be a bit low on a large scale at Northrop, so such blunders are no surprise to me.
at least they never said ... (Score:1, Interesting)
That "they'll fix it so it never happens again."
I used to work in IT and I hated it whenever some manager said that to a customer. Things fail, they can fail multiple times and different failures can have the same results so the statement that "it'll never happen again" would be a bold faced lie.
Re:I was involved with NG on a project once.... (Score:5, Interesting)
I work for a smaller DoD related company (17k employees, so not that small really) and I hear nothing but horror stories about NG, and have experienced one of them myself. NG originally developed our project but was not allowed to bid on the re-compete where we picked it up. It was bad, but we are fixing it. I really can't say much more than that at the moment.
Re:NGC Culture (Score:2, Interesting)
Whoa anecdote! I've worked for DoD contractors for years (and on different coasts). I've never heard of anyone at Northrop being placed on furlogh so I guess we don't know any of the same employees. In my experience, when there are no contracts, the employer covers your salary until the numbers tell them to start laying people off. If you are an employee they either pay you or lay you off. You don't sit around at home waiting for a call. Thats one of the benefits of a big company. They have enough stuff going to ride the wave and not have to lay people off when someone is done a contract and the next one doesn't start for a week. Their incentive is to keep you because they need to show they have enough people to complete a contract to win it.
Now have 2 data points.
Such blunders are commonplace in any large organization. Someone name an entity of 150k+ people that works well to prove me wrong. With large systems things get complicated. There are a lot of stupid people out there and the bigger your organization, the more likely they'll creep in and screw things up with politics etc. But despite the large management structure there are plenty of smart people around to keep things running. Some things are so big they can only be done by huge groups...with a bunch of slackers riding along.
Re:My Project (Score:3, Interesting)
A couple of months ago there was a declassified document (either on /. or Wikileaks) that was basically a CIA/War Department handbook for (civilian) saboteurs against an oppressive regime. If you read this, you can see a lot of the practices that are recommended to be used against the 'enemy' are exactly the same as those contractors use. The goal is to have the enemy (or the company you're contracted to) spend as much money as possible on lost time and resources. The only difference is the people/group that profits from it.
I've been a contractor before for some of those contracting companies - they basically try to keep you as long as possible in a position even if it's totally unnecessary since they get paid ~80%-120% of your (before-tax) wages.
Re:$2.6 billion service contract? (Score:1, Interesting)
I'm seeing the beginning stages of the same type of things in the State of California. Much of the infrastructure is segmented, with each agency operating much of their own IT equipment and services. Some people have come in trying to consolidate IT operations, move all agency equipment to central data centers, centralize email, etc, etc. They aren't saying this, but we know this will eventually mean much of the IT staff from each agency will just be laid off, which could possibly result in the same thing you mention: Paying less in direct employee salaries/pensions, higher cost due to contractors, and lack of knowledge from contractors because all of the employees who set it up are gone. Especially with all of the budget problems here, they are trying to tout it as a way to save costs.
So much of the state's infrastructure is segmented, very old, complex, and undocumented, so good luck to them for trying that. They are still running COBOL on their payroll system for christs sake.
Re:My Project (Score:2, Interesting)
I *was* a NG subcontractor, doing sysadmin work in a datacenter in a different state. Oh, the stories I could tell. More than once, as a new project was in the bidding stage, I would point out serious, game-stopping flaws in the proposal. The response was inevitably along the lines of "Fixing that would require us to raise our bid, and we don't have any exposure this way because we can blame the agency for a flawed proposal. Then they would win the bid, a bunch of equipment would get dumped on the floor, then sit idle for a few months as everyone tried to figure out how to get around a problem that should have been dealt with up front. Coming from the private sector, where satisfying the customer with a working solution was *the* criterion for success, this made my skin crawl. Since I wouldn't play ball, and was always completely candid with my customers about all the shenanigans, guess who's contract wasn't renewed :). Think NG is being unfairly singled out? The very first problem I had to deal with was frequent network outages. Turns out their resident network guy was making "10-base-T" cables with flat (untwisted) cables. I actually had to explain why this was a bad idea, and it took a good while to convince him. (posting AC for obvious reasons).
My experience as a contractor (Score:3, Interesting)
Many places screw over both their clients and their employees.
Re:It was EMC storage failure (Score:3, Interesting)