New Virginia IT Systems Lack Network Backup 211
1sockchuck writes "Virginia's new state IT system is experiencing downtime in key services because of a mind-boggling oversight: the state apparently neglected to require network backup in a 10-year, $2.3 billion outsourcing deal with Northrop Grumman. The issue is causing serious downtime for state services. This fall the Virginia DMV has suffered 12 system outages spanning a total of more than 100 hours, and downtime hampered the state transportation department when a state of emergency was declared during the Nov. 11 Northeaster."
Blame Northrop? (Score:4, Insightful)
Easy (Score:5, Insightful)
During the first six months of the year, state Department of Transportation workers faced 101 significant IT outages totaling 4,677 hours: an average of more than 46 hours per outage. One took 360 hours to fix.
That's 27 weeks of downtime in the space of 26 weeks, which raises a much more important question than why there's no network redundancy and that question is: What kind of fucking morons have they got running their systems?
outsourcing (Score:5, Insightful)
But I thought the magic pixie dust of free enterprise would make outsourcing something to the private sector cheaper, more efficient, and better in every possible way?
Re:Blame Northrop? (Score:5, Insightful)
Likely they were told they should have a backup, quoted a price, and said nah, we will be fine.
Re:Blame Northrop? (Score:4, Insightful)
The companies where I've seen this basically do a risk assessment and say "well, we are willing to accept the risk of downtime because BCP is too costly". Unfortunately they don't weigh the chance of an outage or disaster appropriately, and then find themselves severely screwed when a tornado, storm system, or fire occurs, and then they are either out of business (in a small company) or take enough of a hit to make a headline on Slashdot and cripple the business.
Seriously, when are companies going to realize that this is a critical component of IT? I've felt like I've talked till I was blue in the face about this over the years.
Re:Blame Northrop? (Score:5, Insightful)
Why did Northrop, which presumably has experience in government systems, not design backups?
Because they didn't have to. It wasn't in the contract, so they're not going to spend the money doing it. They're not in business to keep the state government afloat, their only purpose is to make money.
If you don't properly specify your needs, that's your fault. Don't rely on corporate good will, because there is no such thing.
Re:That's the way of the future... (Score:3, Insightful)
Actually, there are plenty of backup systems in Star Trek. Of course, a few of them fail in every episode to avoid having every episode end with a "Yay for Starfleet engineers!" after five minutes.
In fact, for some systems they apparently have up to four backups which all manage to fail magically at the same time *cough*transporters*cough*.
/Mikael
Re:Blame Northrop? (Score:2, Insightful)
Northrop Grumman outsources part of it's own IT as (Score:3, Insightful)
Northrop Grumman outsources part of it's own IT as well and it does not own some of it's systems they rent them or at least they did 1-2 years ago.
Re:Blame Northrop? (Score:3, Insightful)
That'd be my guess. Second guess would be that they agreed to having a backup - as soon as some politician determined where the backup site would be. (Which, of course, hasn't happened yet.)
Re:They have bigger problems than just this one... (Score:4, Insightful)
Bureaucracy is bureaucracy. [itbusiness.ca] Government involvement doesn't mean ineptitude, and the free market doesn't gurantee competence. Whether private or public, ineptitude as well as competence abounds.
Comment removed (Score:5, Insightful)
Re:Blame Northrop? (Score:3, Insightful)
Welcome to the world of government low-bid contracts. The specification didn't call for backups, so you don't get backups, because that would've made the bid higher.
Re:Blame Northrop? (Score:4, Insightful)
True enough. But as you say, Northrop is in the business of making money, so it would've made sense for them to do the following:
* Deliver a offer for the system requested.
* Get the deal signed
* Say: We notice you've not specified any backup, do you want that additionally ?
Gives them a chance to upsell, AND potentially makes the customer happier -- a win-win.
Re:outsourcing (Score:3, Insightful)
The government is clearly involved here. So it's got nothing to do with free enterprise.
Funny math or multiple systems? (Score:4, Insightful)
"During the first six months of the year, state Department of Transportation workers faced 101 significant IT outages totaling 4,677 hours: an average of more than 46 hours per outage. One took 360 hours to fix."
wait, 4,677 hours? how could that be? There were 181 days in the first 6 months of this year, that's only 4,344 hours.. there was more downtime on the system than days in it's operational life! (did someone /0 here?)
Outsourced, no thanks... I think I'd rather dig up a Univac I to do work on, at least it would be more reliable
Re:Blame Northrop? (Score:2, Insightful)
I don't buy that it's necessarily the government's fault for not specifying backups.
The customer should only have to say "we need a system that does X, it needs to be up Y% of the time, with an MTTR of no more than Z." They don't know, and shouldn't have to specify technical details. It's up to the provider to design a system that does that.
As another poster mentioned, though, it's quite likely that NG came back and said here's a system that will do that, and it will cost X, and the customer got sticker shock and decided to drop a few 9s from the SLA. I'm in that business, and this happens all the time.
Re:They have bigger problems than just this one... (Score:1, Insightful)
But private ineptitude tends to be a self-correcting problem.
And this helps me after my health records (or credit card numbers or whatever) have been leaked... how?
One inept company goes bankrupt, another takes its place, and the damage is already done irregardless.
Re:Blame Northrop? (Score:5, Insightful)
You must not deal with the government much :)
If you are bidding for a government contract, it's a public bid. They state their requirements very precisely, and every single dollar you spend over is counted against you.
Basically to do network backup, you'd have to eat it out of the goodness of your heart. There is a potential to upsell later, of course, but it has to go back through the public approvals process.
Re:Blame Northrop? (Score:4, Insightful)
> Sorry, too expensive. We have weighed the risk, and decided it's an acceptable risk.
Yes because they can count on waking you up to fix it.
So seems perhaps the bosses are doing the right thing for the organization. They hired you, you will wake up to fix it, and they don't need to spend on HA/failover.
Now if they hired someone who can't fix it fast, or sleeps really soundly, then they should spend on HA/failover, or hire you instead
Re:Blame Northrop? (Score:5, Insightful)
They're not in business to keep the state government afloat, their only purpose is to make money.
I hate when this is offered as an excuse for shoddy work. "It's not their job to do good work. It's their job to make money." Yeah? So what. It strikes me a little like saying, "Hey, can't blame a con man for stealing your money. That's what con men do!"
I don't know this particular situation well enough to say who is at fault and to what degree, but it's part of their business to service their customers well. It's part of every company's business to provide service to their customers in an ethical manner.
Re:Funny math or multiple systems? (Score:3, Insightful)
4,677 hours of failure in 4,344 hours of time means that at any given time, an average of 1.07 locations were offline.
There are 131 DMV offices in Virignia; I don't know how many other Department of Transportation locations are included in the same bucket. If we assume that it's *only* the 131 DMV offices, 1.07 failures at any given time means that the system means that 130.3 locations are working, meaning that this statewide patchwork of network connections is 99.45% reliable.
If your 'redundant' connections cut the failures in half (which they wouldn't), you'd have 99.59% reliability at more than twice the cost for the network.
Adding 'redundancy' would more than double the network cost (since presumably currently they're using the lowest bidder), and in most places it wouldn't add any real redundancy anyway. Getting actual network redundancy is *fiendishly* difficult, even when you're spending a lot of money and siting a facility in a place that's well-served for networking. In small-town Virginia, you're almost certainly going to wind up paying for having redundant wires hanging on the same poles.
central planning is equally idiotic (Score:3, Insightful)
as free market fundamentalists
it is a hallmark of the triumph of your fear over your intellect that you think that's what i am advocating for
examples of fear triumphing over intellect:
"gay marriage should be legal"
reaction:
"why do you think pedophiles should be allowed to marry boys and bestiality practioners to marry animals?"
#2:
"marijuana should be legal"
reaction:
"why do you want to legalize methamphetamine and heroin?"
#3:
"the government needs to regulate the market in order for it to be stable"
reaction:
"why do you want communist central planning"
do you see the hysteria at work in these examples?
in the future, i suggest you react to what i am actually saying, rather than projecting your irrational fears onto what i am saying, and reacting to that in hysteria
fact: an unregulated marketplace bubbles and pops due to nothing but simple human psychology, and naturally degenerates into a few powerful players dominating everyone else. without regulation, there is no such thing as a "free" market. a stable free market of equals, without regulation by some entity, is something that has never existed in the history of humanity. its a myth cooked up by libertarian fundamentalists, their garden of eden. its a fantasy of blindness in direct contradiction to obvious well-established human behavior:
1. people will take advantage of others, take advantage of natural imperfections in the market, and establish domination and exploitation of later arrivals to the marketplace
2. people will react in panic and fear at rumors, and destroy the market on nothing but emotion: calm rational decisions does not dominate the market
do you care to defy these simple obvious truths?
Re:there's a lot of obstinance in your comment (Score:3, Insightful)
1. people will take advantage of others, take advantage of natural imperfections in the market, and establish domination and exploitation of later arrivals to the marketplace
2. people will react in panic and fear at rumors, and destroy the market on nothing but emotion: calm rational decisions does not dominate the market
do you deny either simple obvious truth?
how do you fight #1, and #2 then?
answer: you need the government to regulate it. duh
an unregulated marketplace bubbles and pops due to nothing but simple human psychology, and naturally degenerates into a few powerful players dominating everyone else. without regulation, there is no such thing as a "free" market. a stable free market of equals, without regulation by some entity, is something that has never existed in the history of humanity. its a myth cooked up by libertarian fundamentalists, their garden of eden. its a fantasy of blindness in direct contradiction to obvious well-established human behavior
this has been today's intellectual charity offering for you. do try to take advantage of the offering, and accept the fucking obvious for once in your life about this subject matter
you cannot have a rational discussion (Score:3, Insightful)
with someone who has seriously embraces an absurd premise: that markets left to their own devices are stable and egalitarian
markets left to their own devices bubble and pop, and are manipulated and dominated by entrenched insiders
to not understand this is equivalent to someone refusing to accept that that the sun rises and sets or that the tides go up and down. how can you have a rational discussion with someone who refuses to see and accept obvious factual aspects of the reality they live in?