State of Virginia Technology Centers Down 190
bswooden writes "Some rather important departments (DMV, Social Services, Taxation) in the state of Virginia are currently without access to documents and information as a technology meltdown has caused much of their infrastructure to be offline for over 24 hours now. State CIO Sam Nixon said, 'A failure occurred in one memory card in what is known as a "storage area network," or SAN, at Virginia's Information Technologies Agency (VITA) suburban Richmond computing center, one of several data storage systems across Virginia.' How does the IT for some of the largest departments in a state come to a screeching halt over a single memory card? Oh, and also, the state is paying Northrup Grumman $2.4 billion over 10 years to manage the state's IT infrastructure."
Reader miller60 adds, "Virginia's IT systems drew scrutiny last fall when state agencies reported rolling outages due to the lack of network redundancy."
Re:card? (Score:4, Informative)
Re:Northrup Grumman (Score:1, Informative)
Northrop Grumman, to some of the other contractors, is also known to be a screw-up that puts out mediocre quality work with a high price tag.
LMT, Raytheon, SAIC, and Honeywell would have all been better choices for making quality products.
Re:Question. (Score:4, Informative)
Well, as Sherlock Holmes' greatest axiom goes "When you have eliminated the impossible, whatever remains, however improbable, must be the truth." Using that logic, the answer is simple. They're not using a SAN. Somewhere along the line someone is bullshitting, and my gut tells me its management. A lot of folks who get government contracts pretty much view them as an opportunity to skim off the top. Why, take what should be a $50,000 solution and mock something up for $10,000, and that's $40,000 profit.
Re:card? (Score:3, Informative)
Re:HA fail (Score:3, Informative)
Sweet Zombie Jesus.
If the RAM in our 8TB Netgear SAN fries it doesn't blow up my office, what the hell are they and Northrup Grumman doing?
It happens (Score:4, Informative)
This is a clear example of the malfeasance that happens when government gets corrupted by corporate interests. Taxpayers in VA should be up in arms about this one.
Here's my story of state agency screw-ups. Two jobs ago I was working for the Secretary of State's office here. We had the opportunity and funding to get our IT infrastructure in order when the Help America Vote Act (HAVA) became law. We were able to build out a secure and redundant room to house our critical infrastructure.
Physical access by key and alarm code only, Redundant power which included an APS Symmetra UPS system, backed up by a 125kW natural gas fired generator. Even made sure to extend tendrils from the redundant power out to the MDF so the ISP could use our power system. Also had redundant cooling tied to the generator.
The one Achilles Heel of the operation was DNS. Ours was provided from outside our space.Suggested they build a zone locally that way we'd have DNS services if the state's went down. But they quashed it as being too difficult! Ut si!
Well one day there's a massive power outage in the city. They were still up and running, lights on, air conditioning on but couldn't get in or out of the internal network even though the ISP circuits were still up. Yup, DNS!
Re:Typical Republican Corruption (Score:1, Informative)
Actually, from the article, you would have seen that the contract was signed back in 2005, when Virginia enjoyed the presence of Mark Warner, Democrat, and now US Senator for Virginia.
Amusingly, Aneesh Chopra, the current CTO of the Obama administration, was the Virginia Secretary of Technology starting shortly after this was signed, and he never added redundancy to the service contract. This was during Warner's tenure and during Tim Kaine's (D-Va) tenure [datacenterknowledge.com].
Also, counter to your argument, it was actually Bob McDonnell, the current Republican Governor, that renegotiated the contract to include redundancy [govtech.com].
With all of that said, I do not think Northrop Grumman was the best fit for this job and after so many egregious failures, they deserve to have their contract reworked in VA's favor, but bureaucracy being what it is, regardless of party politics, I doubt this will change. I really feel like this kind of contract could have gone to a small-to-medium sized VA business that could have handled it extremely well, and locally, for much less. The real sad thing is that the guy who's largest job was to oversee this contract, and did nothing, is now the CTO for the entire country. I don't care what party you are, that's a scary thought.
Re:What brand? (Score:1, Informative)
The following document contains a detailed inventory of the Storage systems used by them, and we can presume it was one of these that failed.
Full details of the contracts also enclosed.
http://www.google.com/url?url=http://www.vita.virginia.gov/procurement/contracts/docs/abstracts/va-040330-stk.pdf&rct=j&sa=X&ei=yfl3TOOLCI-54gbuteG0Bg&ved=0CBkQzgQoADAA&q=Virginia+Information+Technologies+Agency+hp+storage&usg=AFQjCNGRQFneR1OJXIxQGO0mYbizW67bow
Lol.
Re:It's always money (Score:4, Informative)
"get 100 units and run them for 6 months..."
Which works if you presume a linear fail rate, which is bonkers. Systems always run better at the beginning of their lifecycle. Static buildup, electrical interference, repeated heating and cooling cycles, etc all take a toll on the electronics. Would you really personally estimate a real-world MTBF of off-the-shelf SATA drives at 70 years? No, because they work perfectly well for the first year, start having trouble the second, and are all dead by the 8th. But if you presume linear dropoff using just that first year of testing, they look pretty damn bomb proof because that's when they work best. It's a stupid system that's only valid if you replace all of your hardware every year.
And all systems have moving parts. Electrons move. The circuit boards expand and contract. Crap builds up on important components. Electroplating can move move metals from one part of the design to another. Stuff gets plugged in and unplugged.
I realize that MTBF has a very technical definition that is different than marketing departments utilize it as. I might agree with you that any engineer worth their salt can extrapolate a proper MTBF. But most of the MTBF's I've seen are just stupidly wrong. If people really believe those published fantasy numbers, no wonder they don't put enough redundancy in their systems.
Not the only failure VITA has had (Score:1, Informative)
They tried to do a full renumbering of the state's IP address space. This has morphed into a MPLS rollout when it turned out that too much was breaking as they moved various offices around.
State workers hate the whole VITA idea - it has been nothing but disaster and failure since it started.
I love the idea that my tax dollars have been funding this clusterfuck.