7232468
story
Comments: 211 +- IT: New Virginia IT Systems Lack Network Backup on Tuesday November 24, @08:17AM
background: url(//a.fsdn.com/sd/topics/topiccommunications.gif); width:96px; height:62px;
communications
background: url(//a.fsdn.com/sd/topics/topicgovt.gif); width:57px; height:80px;
government
background: url(//a.fsdn.com/sd/topics/topicnetworking.gif); width:71px; height:37px;
networking
background: url(//a.fsdn.com/sd/topics/topicit.gif); width:75px; height:61px;
it
1sockchuck writes "Virginia's new state IT system is experiencing downtime in key services because of a mind-boggling oversight: the state apparently neglected to require network backup in a 10-year, $2.3 billion outsourcing deal with Northrop Grumman. The issue is causing serious downtime for state services. This fall the Virginia DMV has suffered 12 system outages spanning a total of more than 100 hours, and downtime hampered the state transportation department when a state of emergency was declared during the Nov. 11 Northeaster."
Related Stories
It doesn't much signify whom one marries, for one is sure to find out
next morning it was someone else.
-- Will Rogers
All trademarks and copyrights on this page are owned by their respective owners. Comments are owned by the Poster. The Rest © 1997-2010 Geeknet, Inc.
Blame Northrop? (Score:4, Insightful)
Re:Blame Northrop? (Score:5, Insightful)
Likely they were told they should have a backup, quoted a price, and said nah, we will be fine.
Parent
Re: (Score:3, Insightful)
That'd be my guess. Second guess would be that they agreed to having a backup - as soon as some politician determined where the backup site would be. (Which, of course, hasn't happened yet.)
Re:Blame Northrop? (Score:5, Informative)
... as soon as some politician determined where the backup site would be. (Which, of course, hasn't happened yet.)
Actually, it has happened. The CoVA backup site is located in Lebanon, VA (SW part of the state).
What THIS article is discussing is the lack of network backup, not data backup.
This is an important distinction, to say the least.
Parent
Re:Blame Northrop? (Score:4, Insightful)
The companies where I've seen this basically do a risk assessment and say "well, we are willing to accept the risk of downtime because BCP is too costly". Unfortunately they don't weigh the chance of an outage or disaster appropriately, and then find themselves severely screwed when a tornado, storm system, or fire occurs, and then they are either out of business (in a small company) or take enough of a hit to make a headline on Slashdot and cripple the business.
Seriously, when are companies going to realize that this is a critical component of IT? I've felt like I've talked till I was blue in the face about this over the years.
Parent
Re: (Score:3, Interesting)
You are correct. There should be SLAs, as well. The problem that most people don’t seem to comprehend. NG’s contract is NOT with the Stateits with VITA (yea a state agency..). NG does have SLA’s with VITA however most state agencies didn’t even SEE those SLA’s until oh I don’t know the last 3 months? VITA on the other hand, whom state agencies are REQUIRED to use by state law, has NO SLA’s..no MOU (memorandum of Understanding) nothing with the other state agenc
Re:Blame Northrop? (Score:4, Insightful)
> Sorry, too expensive. We have weighed the risk, and decided it's an acceptable risk.
Yes because they can count on waking you up to fix it.
So seems perhaps the bosses are doing the right thing for the organization. They hired you, you will wake up to fix it, and they don't need to spend on HA/failover.
Now if they hired someone who can't fix it fast, or sleeps really soundly, then they should spend on HA/failover, or hire you instead
Parent
Re:Blame Northrop? (Score:5, Insightful)
Why did Northrop, which presumably has experience in government systems, not design backups?
Because they didn't have to. It wasn't in the contract, so they're not going to spend the money doing it. They're not in business to keep the state government afloat, their only purpose is to make money.
If you don't properly specify your needs, that's your fault. Don't rely on corporate good will, because there is no such thing.
Parent
Re:Blame Northrop? (Score:4, Insightful)
True enough. But as you say, Northrop is in the business of making money, so it would've made sense for them to do the following:
* Deliver a offer for the system requested.
* Get the deal signed
* Say: We notice you've not specified any backup, do you want that additionally ?
Gives them a chance to upsell, AND potentially makes the customer happier -- a win-win.
Parent
Re: (Score:2)
Re: (Score:2, Insightful)
I don't buy that it's necessarily the government's fault for not specifying backups.
The customer should only have to say "we need a system that does X, it needs to be up Y% of the time, with an MTTR of no more than Z." They don't know, and shouldn't have to specify technical details. It's up to the provider to design a system that does that.
As another poster mentioned, though, it's quite likely that NG came back and said here's a system that will do that, and it will cost X, and the customer got sticker sho
Re:Blame Northrop? (Score:5, Insightful)
You must not deal with the government much :)
If you are bidding for a government contract, it's a public bid. They state their requirements very precisely, and every single dollar you spend over is counted against you.
Basically to do network backup, you'd have to eat it out of the goodness of your heart. There is a potential to upsell later, of course, but it has to go back through the public approvals process.
Parent
Re:Blame Northrop? (Score:5, Insightful)
They're not in business to keep the state government afloat, their only purpose is to make money.
I hate when this is offered as an excuse for shoddy work. "It's not their job to do good work. It's their job to make money." Yeah? So what. It strikes me a little like saying, "Hey, can't blame a con man for stealing your money. That's what con men do!"
I don't know this particular situation well enough to say who is at fault and to what degree, but it's part of their business to service their customers well. It's part of every company's business to provide service to their customers in an ethical manner.
Parent
Re: (Score:2, Insightful)
Re: (Score:2)
How do you know that Northrup didn't suggest them? Or that Northrop wasn't the lowest qualified bidder by being the only one not to include backups in their bid? When you buy a computer do you expect the vendor to throw in a free backup system even though you didn't ask for it "because you should have it"?
Sounds to me like the state got what it ordered.
Re: (Score:2)
In my experience, it is rare for a customer, even with professional IT staff, to properly specify their needs when it comes to technology. Why did Northrop, which presumably has experience in government systems, not design backups?
Consultants never seem to get it right. Granted, the situations I'm familiar with are a fraction of a fraction of a percent of the size of this contract but at that scale the issue is that the contractors and consultants are juggling multiple clients so the needs of any one must be balanced against the needs of all of them. But what really burns me is when they can't even provide decent advice on what should be bread and butter. "We need a backup solution." See, there you go. Many solutions on the market bu
Re: (Score:3, Insightful)
Welcome to the world of government low-bid contracts. The specification didn't call for backups, so you don't get backups, because that would've made the bid higher.
That's the way of the future... (Score:5, Funny)
Have you ever seen backup systems in Star Trek, for example? you haven't. The future requires no backups.
Re: (Score:3, Insightful)
Actually, there are plenty of backup systems in Star Trek. Of course, a few of them fail in every episode to avoid having every episode end with a "Yay for Starfleet engineers!" after five minutes.
In fact, for some systems they apparently have up to four backups which all manage to fail magically at the same time *cough*transporters*cough*.
/Mikael
Re: (Score:2)
Talking about that, is there a single instance in Star Trek where the "Manual Override" actually worked?
And then, how is it a manual override if you just flip some other switches. The way they use "manual overrides" in Star Trek the bridge should be similar to that of the Tardis.
Re: (Score:2)
Actually, yes. In one episode of NG they had to purge the system of malware and indeed used their backup. The only time I ever saw a backup where I work was when it was part of my job to change the tapes.
Easy (Score:5, Insightful)
During the first six months of the year, state Department of Transportation workers faced 101 significant IT outages totaling 4,677 hours: an average of more than 46 hours per outage. One took 360 hours to fix.
That's 27 weeks of downtime in the space of 26 weeks, which raises a much more important question than why there's no network redundancy and that question is: What kind of fucking morons have they got running their systems?
Re: (Score:2)
What kind of downtime are we talking? There's a big difference between a 5 hour total outage or 5 hour loss of connectivity to a small town DMV office.
PHB who say it's cheaper to let some out side peop (Score:2)
PHB's who say it's cheaper to let some out side people run there IT and likely say no to the higher cost plan that has backup with it.
outsourcing (Score:5, Insightful)
But I thought the magic pixie dust of free enterprise would make outsourcing something to the private sector cheaper, more efficient, and better in every possible way?
Re:outsourcing (Score:4, Informative)
Hey, it worked. Mark Warner won two-thirds of the vote in his senate run last year based on his stellar performance as governor. This was one of his big initiatives.
(He also *fixed* the revenue sources, so that there'd never be a problem like happened with Jim Gilmore. Yet, now, Virginia is in worse shape than when he got there.)
Parent
Re: (Score:3, Insightful)
The government is clearly involved here. So it's got nothing to do with free enterprise.
"the government exists (Score:5, Informative)
and has involved itself in the market in some way in the past
therefore, any prudent rational criticism of the free market and how it obviously fails can be explained away with creative rationalization that its the government's fault, somehow"
my favorite is how free market fundamentalists wish to blame the market crash of 2008 on government policies. rather than gee, i dunno, the clinton and bush administration deregulation policies? you know, deregulation: having the government less invovled int he market?
"what? my free market bubble and pop? nah, impossible! government's fault! pffft"
please study your banking panics of the 1800s: without regulation, free markets have innate imperfections which always result in catalcysmic failures. all you need is simple human psychology, no government need apply, to cause a market to crash. you either regulate it, leveling the playing field artificially, and therefore making it truly "free", or you leave it alone, letting it bubble and pop like mad, and allow monopolists to take advantage of natural imperfections in the market to leverage unfair behavior
free market fundamentalism is dead. your ideology is dead. fact: you need government involvement in the market for the market to run efficiently. fact: you need government policing and regulation of the marketplace to keep it "free" and egalitarian and equal for all players
if you don't understand these simple truths by now, or refuse to believe that despite the obvious proof, you're an idiot
Parent
central planning is equally idiotic (Score:3, Insightful)
as free market fundamentalists
it is a hallmark of the triumph of your fear over your intellect that you think that's what i am advocating for
examples of fear triumphing over intellect:
"gay marriage should be legal"
reaction:
"why do you think pedophiles should be allowed to marry boys and bestiality practioners to marry animals?"
#2:
"marijuana should be legal"
reaction:
"why do you want to legalize methamphetamine and heroin?"
#3:
"the government needs to regulate the market in order for it to be stable"
reaction:
"
Re: (Score:3, Insightful)
1. people will take advantage of others, take advantage of natural imperfections in the market, and establish domination and exploitation of later arrivals to the marketplace
2. people will react in panic and fear at rumors, and destroy the market on nothing but emotion: calm rational decisions does not dominate the market
do you deny either simple obvious truth?
how do you fight #1, and #2 then?
answer: you need the government to regulate it. duh
an unregulated marketplace bubbles and pops due to nothing but si
you're a free market fundamentalist (Score:3, Informative)
you believe the market left alone takes care of itself, and the government makes it unstable
this is the opposite of reality: a free market is inherently unstable. government involvement stabilizes it
i'm sorry i don't have any books by crackpots to cite to prop up the fucking obvious truth for you
not like you would accept it
you cannot have a rational discussion (Score:3, Insightful)
with someone who has seriously embraces an absurd premise: that markets left to their own devices are stable and egalitarian
markets left to their own devices bubble and pop, and are manipulated and dominated by entrenched insiders
to not understand this is equivalent to someone refusing to accept that that the sun rises and sets or that the tides go up and down. how can you have a rational discussion with someone who refuses to see and accept obvious factual aspects of the reality they live in?
what? (Score:3, Funny)
no supplemental reading material from libertarian crackpots about the virginal holiness of the unregulated market?
Re: (Score:2)
Re: (Score:2)
The best way to do that is to do the best job for the best price. That's why something "smells" here..
If you'll excuse my saying so, that's an incredibly naive way of seeing things.
IME there are lots of business plans which history has shown can all work very well indeed.
Off the top of my head, there's:
They have bigger problems than just this one... (Score:4, Informative)
Remember how Virginia's health records were compromised earlier this year?
http://it.slashdot.org/article.pl?sid=09/05/05/1232240 [slashdot.org]
Sounds like systemic ineptitude which is why I'm really looking forward to more government involvement in health care!
Re:They have bigger problems than just this one... (Score:4, Funny)
Parent
Re:They have bigger problems than just this one... (Score:4, Insightful)
Bureaucracy is bureaucracy. [itbusiness.ca] Government involvement doesn't mean ineptitude, and the free market doesn't gurantee competence. Whether private or public, ineptitude as well as competence abounds.
Parent
Re: (Score:2)
That's often true. But private ineptitude tends to be a self-correcting problem. Businesses that are consistently unresponsive to the market and that do really stupid things will fail. And they should fail. Better businesses spend a great deal of effort to identify their mistakes so that they won't repeat them.
Public bureaucracies, OTOH, are essentially pure monopolies. They entrench themselves and always outlive whatever original purpose they had, they're given the power of law to enforce their decisions,
Re: (Score:2)
Remember how Virginia's health records were compromised earlier this year?
http://it.slashdot.org/article.pl?sid=09/05/05/1232240 [slashdot.org]
Sounds like systemic ineptitude which is why I'm really looking forward to more government involvement in health care!
Yeah, right. Because the private sector has done such a good job of protecting our privacy, banking info, etc.
Please..., go troll somewhere else.
NG, I call you out! (Score:4, Interesting)
Most forget that the network provided by the NG crooks is NOT part of the Comprehensive Infrastructure Agreement (CIA). It is a seperate agreement that is a fixed cost agreement under which NG was supposed to replace “like for like”. They were supposed to install an MPLS network. MPLS (Multi Protocol Label Switching) allows for the prioritization of traffic to allow Voice traffic to travel over the same circuit as the data. It also supposed to be intelligent enough to encrypt data to essentially allow a VPN to be created from point-to-point.
None of the VPN has been done as promised, very few sites have used the VOIP option unless dictated to by VITA as part of new construction and most sites complain about network performance. Some agencies had totally redundant networks but were forced to pay more for less. 65% of VITA staff make over 90,000 a year. Again we pay more for less.
While I am not a NG fan, interestingly enough, most state managers at Agencies will tell you that working directly with NG allows things to get done, VITA just gets in the way. VITA wants to always be the interface, Waste Fraud and Abuse to pay high salaries for mostly unqualified folks. Throw out VITA and let the agencies be treated like customers by NG.
The IT Community Frowns Upon Your Shenanigans...
Northrop Grumman outsources part of it's own IT as (Score:3, Insightful)
Northrop Grumman outsources part of it's own IT as well and it does not own some of it's systems they rent them or at least they did 1-2 years ago.
Network connections, not system backups... (Score:5, Informative)
Network redundancy not backups (Score:5, Insightful)
The article does not mention "backups" as in tape drives and off-site storage.
The article does mention lack of redundancy at the network carrier level.
My guess is that Northrop Grumman designed a network around single circuits connecting offices to data centers, and did not design the network to tolerate WAN link failures.
A stupid oversight for sure, but nothing that can't be easily remedied by ordering redundant WAN circuits from your telco of choice. Redundant routing gear would also be smart.
For all that are blaming government for this - they outsourced the design and implementation to a private company. That company screwed the pooch in design and implementation. Shame on both parties for not recognizing the risk of WAN failure.
-ted
Epic Fail (Score:2, Interesting)
If any story deserves this tag it is this. from the article:
"Virginia declared a state of emergency Nov. 11 in the face of record nor'easter rains and winds.
But without backup circuits -- which VDOT had before the Northrop Grumman outsourcing -- to take up the load, the transportation agency's Hampton Roads' IT network went out of service 23 times during the event.
"We called at 5:35 in the morning," said Gary Allen, VDOT's chief of technology, research and innovation.
"It took VITA four hours to open the hel
Funny math or multiple systems? (Score:4, Insightful)
"During the first six months of the year, state Department of Transportation workers faced 101 significant IT outages totaling 4,677 hours: an average of more than 46 hours per outage. One took 360 hours to fix."
wait, 4,677 hours? how could that be? There were 181 days in the first 6 months of this year, that's only 4,344 hours.. there was more downtime on the system than days in it's operational life! (did someone /0 here?)
Outsourced, no thanks... I think I'd rather dig up a Univac I to do work on, at least it would be more reliable
Re: (Score:2, Informative)
My guess is that would be multiple systems. They noted in TFA that they provided IT services to 1000 local governments and 85 state agencies in VA.
Re: (Score:3, Insightful)
4,677 hours of failure in 4,344 hours of time means that at any given time, an average of 1.07 locations were offline.
There are 131 DMV offices in Virignia; I don't know how many other Department of Transportation locations are included in the same bucket. If we assume that it's *only* the 131 DMV offices, 1.07 failures at any given time means that the system means that 130.3 locations are working, meaning that this statewide patchwork of network connections is 99.45% reliable.
If your 'redundant' connectio
THIS explains all the closed rest stops in VA... (Score:3, Interesting)
I *knew* there had to be some other reason why they closed half the interstate rest stops in VA, this is obviously where the money was (mis)spent...
15 day outage??? (Score:4, Funny)
From TFA:
Suddenly, I don't feel so bad for that 2 1/2 hour glitch last week :)