New Virginia IT Systems Lack Network Backup 211
1sockchuck writes "Virginia's new state IT system is experiencing downtime in key services because of a mind-boggling oversight: the state apparently neglected to require network backup in a 10-year, $2.3 billion outsourcing deal with Northrop Grumman. The issue is causing serious downtime for state services. This fall the Virginia DMV has suffered 12 system outages spanning a total of more than 100 hours, and downtime hampered the state transportation department when a state of emergency was declared during the Nov. 11 Northeaster."
NG, I call you out! (Score:4, Interesting)
Most forget that the network provided by the NG crooks is NOT part of the Comprehensive Infrastructure Agreement (CIA). It is a seperate agreement that is a fixed cost agreement under which NG was supposed to replace “like for like”. They were supposed to install an MPLS network. MPLS (Multi Protocol Label Switching) allows for the prioritization of traffic to allow Voice traffic to travel over the same circuit as the data. It also supposed to be intelligent enough to encrypt data to essentially allow a VPN to be created from point-to-point.
None of the VPN has been done as promised, very few sites have used the VOIP option unless dictated to by VITA as part of new construction and most sites complain about network performance. Some agencies had totally redundant networks but were forced to pay more for less. 65% of VITA staff make over 90,000 a year. Again we pay more for less.
While I am not a NG fan, interestingly enough, most state managers at Agencies will tell you that working directly with NG allows things to get done, VITA just gets in the way. VITA wants to always be the interface, Waste Fraud and Abuse to pay high salaries for mostly unqualified folks. Throw out VITA and let the agencies be treated like customers by NG.
The IT Community Frowns Upon Your Shenanigans...
Re:Blame Northrop? (Score:3, Interesting)
You are correct. There should be SLAs, as well. The problem that most people don’t seem to comprehend. NG’s contract is NOT with the Stateits with VITA (yea a state agency..). NG does have SLA’s with VITA however most state agencies didn’t even SEE those SLA’s until oh I don’t know the last 3 months? VITA on the other hand, whom state agencies are REQUIRED to use by state law, has NO SLA’s..no MOU (memorandum of Understanding) nothing with the other state agencies. We are FORCED to use VITAand they have no staff, and they know that if and when it fails, NG will be blamed.
NG has failed and will continue to fail not because “agencies want us to fail” but because they came in with no clear understanding and NO desire to understand state governments business or needs. We are NOT a private company, we are NOT driven by profit. We are stymied by laws and procedures to protect the tax payer that in the end cost more money.
State workers just want to get this thing working. WE KNOW how its suppose to workwe’ve done it. We’ve done it many years without budget money, without staff, without support.
This VITA/NG “parnership” is a complete fiasco for those of us that live it every day. This is just the tip of the iceburg.
Epic Fail (Score:2, Interesting)
If any story deserves this tag it is this. from the article:
"Virginia declared a state of emergency Nov. 11 in the face of record nor'easter rains and winds.
But without backup circuits -- which VDOT had before the Northrop Grumman outsourcing -- to take up the load, the transportation agency's Hampton Roads' IT network went out of service 23 times during the event.
"We called at 5:35 in the morning," said Gary Allen, VDOT's chief of technology, research and innovation.
"It took VITA four hours to open the help ticket" and begin to solve the problem."
4 hrs on a critical system seriously just to get started solving it?
THIS explains all the closed rest stops in VA... (Score:3, Interesting)
I *knew* there had to be some other reason why they closed half the interstate rest stops in VA, this is obviously where the money was (mis)spent...
Re:Network redundancy not backups (Score:1, Interesting)
I've worked on quite a few projects similar to this and redundancy for ethernet services is the exception rather than the rule.
Very often government grants will specifically mention that they will not pay extra costs associated with redundancy.
Re:Network redundancy not backups (Score:1, Interesting)
Redundant WAN links may help, but most likely you will have 2 downed circuits coming into the building when the local pedestal fills up with water. If you don't want to spend the money to bring in disparate carriers from disparate PoPs through different cable paths, then you should look at a non-terrestrial option like satellite or 3g. Both are inexpensive and provide adequate bandwidth to survive a circuit failure, assuming of course that you're not trying to back up a ds3.
Re:Blame Northrop? (Score:1, Interesting)
At least with the Census Bureau, a contractor doesn't even need to deliver a product to be paid. They will be paid overtime to finish a product that was supposed to be delivered a week ago. In my experience, incompetence pays if you're a contractor. Bid low, get contract, take longer, get paid more. Eventually, the company gets blacklisted so they won't be able to bid in the future, but they can always just form a new company name and start over.