Forgot your password?
typodupeerror
Communications Government Networking IT

New Virginia IT Systems Lack Network Backup 211

Posted by timothy
from the private-did-not-make-perfect dept.
1sockchuck writes "Virginia's new state IT system is experiencing downtime in key services because of a mind-boggling oversight: the state apparently neglected to require network backup in a 10-year, $2.3 billion outsourcing deal with Northrop Grumman. The issue is causing serious downtime for state services. This fall the Virginia DMV has suffered 12 system outages spanning a total of more than 100 hours, and downtime hampered the state transportation department when a state of emergency was declared during the Nov. 11 Northeaster."
This discussion has been archived. No new comments can be posted.

New Virginia IT Systems Lack Network Backup

Comments Filter:
  • Blame Northrop? (Score:4, Insightful)

    by betterunixthanunix (980855) on Tuesday November 24, 2009 @09:23AM (#30212696)
    In my experience, it is rare for a customer, even with professional IT staff, to properly specify their needs when it comes to technology. Why did Northrop, which presumably has experience in government systems, not design backups?
    • Re:Blame Northrop? (Score:5, Insightful)

      by eht (8912) on Tuesday November 24, 2009 @09:26AM (#30212732)

      Likely they were told they should have a backup, quoted a price, and said nah, we will be fine.

      • Re: (Score:3, Insightful)

        by Daniel_Staal (609844)

        That'd be my guess. Second guess would be that they agreed to having a backup - as soon as some politician determined where the backup site would be. (Which, of course, hasn't happened yet.)

        • Re:Blame Northrop? (Score:5, Informative)

          by DRBivens (148931) on Tuesday November 24, 2009 @10:15AM (#30213258) Journal

          ... as soon as some politician determined where the backup site would be. (Which, of course, hasn't happened yet.)

          Actually, it has happened. The CoVA backup site is located in Lebanon, VA (SW part of the state).

          What THIS article is discussing is the lack of network backup, not data backup.

          This is an important distinction, to say the least.

          • by vegiVamp (518171)
            So they're lacking a *backup network*, not a network backup ?
          • by TheLink (130905)
            Yeah, and the problem might be overblown (and the reason why it is overblown could be politics or part of "haggling").

            100 hours in five weeks might be not much depending on how many branches there are. Same goes for the 4,677 hours in 6 months figure.

            The branches may just have ADSL links to the HQ.

            As for redundant network links. In many cases it's worthless paying the extra. It costs a lot to do it right.

            1) The branch may only have one telco/carrier choice for connectivity (at a reasonable price)
            2) Even if
            • by atamido (1020905)

              2) Even if they have two choices, both could go through the same line/chunk of ground that the backhoe digs up. Nowadays with all the outsourcing and inter company deals, you might buy redundant links from two different companies, and later find out the hard way that they are all in the same cable!

              For the small city that I work for, we could get AT&T ADSL, Time Warner Cable, or Time Warner Fiber. They all terminate into the same decades old AT&T building. It is certainly possible that Time Warner has their own fiber running between that AT&T building and the rest of the world, but I'd bet money that they just lease space on whatever data trunk AT&T pulled out there.

              One misplaced backhoe is going to take out data and phone for the entire city, no matter who their provider is.

    • Re:Blame Northrop? (Score:4, Insightful)

      by skgrey (1412883) on Tuesday November 24, 2009 @09:29AM (#30212758)
      And not just backups, it sounds like they had no BCP plan at all. This is a massive oversight, but a fairly common one. I've consulted for a number of years, and it's amazing at how many companies don't have a BCP plan at all, and sometimes it includes simple backups of data.

      The companies where I've seen this basically do a risk assessment and say "well, we are willing to accept the risk of downtime because BCP is too costly". Unfortunately they don't weigh the chance of an outage or disaster appropriately, and then find themselves severely screwed when a tornado, storm system, or fire occurs, and then they are either out of business (in a small company) or take enough of a hit to make a headline on Slashdot and cripple the business.

      Seriously, when are companies going to realize that this is a critical component of IT? I've felt like I've talked till I was blue in the face about this over the years.
      • Re: (Score:3, Interesting)

        by Anonymous Coward

        You are correct. There should be SLAs, as well. The problem that most people don’t seem to comprehend. NG’s contract is NOT with the Stateits with VITA (yea a state agency..). NG does have SLA’s with VITA however most state agencies didn’t even SEE those SLA’s until oh I don’t know the last 3 months? VITA on the other hand, whom state agencies are REQUIRED to use by state law, has NO SLA’s..no MOU (memorandum of Understanding) nothing with the other state agenc

    • Re:Blame Northrop? (Score:5, Insightful)

      by mcgrew (92797) * on Tuesday November 24, 2009 @09:30AM (#30212784) Homepage Journal

      Why did Northrop, which presumably has experience in government systems, not design backups?

      Because they didn't have to. It wasn't in the contract, so they're not going to spend the money doing it. They're not in business to keep the state government afloat, their only purpose is to make money.

      If you don't properly specify your needs, that's your fault. Don't rely on corporate good will, because there is no such thing.

      • Re:Blame Northrop? (Score:4, Insightful)

        by Eivind (15695) <eivindorama@gmail.com> on Tuesday November 24, 2009 @10:04AM (#30213150) Homepage

        True enough. But as you say, Northrop is in the business of making money, so it would've made sense for them to do the following:

        * Deliver a offer for the system requested.
        * Get the deal signed
        * Say: We notice you've not specified any backup, do you want that additionally ?

        Gives them a chance to upsell, AND potentially makes the customer happier -- a win-win.

        • It's a tight budget year. Perhaps VA said no, when the additional work was suggested.
          • by Bakkster (1529253)

            Living in the area, I can tell you VA has definitely been having budget issues. It's also an election year, and the only thing voters wanted their money spent on is roads (one NPR call-in show took about 10 calls for an official, all about the road budget). Put 2 and 2 together, and there's no room in the budget for 'fluff' like a redundant network.

        • by dissy (172727)

          so it would've made sense for them to do the following:

          * Deliver a offer for the system requested.
          * Get the deal signed
          * Say: We notice you've not specified any backup, do you want that additionally ?

          Who's to say they didn't? ;)

          My guess is the disconnect happened in between steps 1 and 2, more precisely when the bean counter saw the price in the offer and mentally could not attach that to any value.

        • "Thank you for your suggestion that we spend more money. We'll certainly take it under consideration."

          --Later:

          "File this somewhere where I'll be sure to never see it again, won't you, Miss Haversham?"

        • by careysub (976506)

          True enough. But as you say, Northrop is in the business of making money, so it would've made sense for them to do the following:

          * Deliver a offer for the system requested. * Get the deal signed * Say: We notice you've not specified any backup, do you want that additionally ?

          Gives them a chance to upsell, AND potentially makes the customer happier -- a win-win.

          According to article:

          In a unique public-private venture, Virginia agreed in 2005 to let the giant defense and information contractor Northrop Grumman run nearly all the state's IT systems.
          The 10-year, $2.3 billion project aims to modernize 85 state government agencies' computer networks, PCs, phones, servers and e-mail systems, while holding down costs. The deal also provides IT services to about 1,000 local government customers.

          This suggests that this was not just a typical IT project on public bid -- No

        • Happens with government contracts all the time.

          McDonnell Douglas vs. Locheed Martin. Often times McDonnell Douglas would create a design that went beyond the spec, including many of those "Well it should have this, or that". Locheed bided the contract spec, nothing more, and in the last few rounds of fighter programs won because they bid the spec. Then when the AF or Navy would come back and say, "Gee it should do XYZ" Locheed would say "Sure, it will be another $XX Million".

          McDonnell Douglas did a revie

      • Re: (Score:2, Insightful)

        by eth1 (94901)

        I don't buy that it's necessarily the government's fault for not specifying backups.

        The customer should only have to say "we need a system that does X, it needs to be up Y% of the time, with an MTTR of no more than Z." They don't know, and shouldn't have to specify technical details. It's up to the provider to design a system that does that.

        As another poster mentioned, though, it's quite likely that NG came back and said here's a system that will do that, and it will cost X, and the customer got sticker sho

        • Re:Blame Northrop? (Score:5, Insightful)

          by WinterSolstice (223271) on Tuesday November 24, 2009 @10:31AM (#30213418)

          You must not deal with the government much :)

          If you are bidding for a government contract, it's a public bid. They state their requirements very precisely, and every single dollar you spend over is counted against you.

          Basically to do network backup, you'd have to eat it out of the goodness of your heart. There is a potential to upsell later, of course, but it has to go back through the public approvals process.

        • by Bakkster (1529253)

          As another poster mentioned, though, it's quite likely that NG came back and said here's a system that will do that, and it will cost X, and the customer got sticker shock and decided to drop a few 9s from the SLA. I'm in that business, and this happens all the time.

          That sounds about right, expecially after reading this quote:

          George F. Coulter took over as the state's chief information officer in August.
          "The first thing I noticed was that the network that Northrop Grumman rolled out didn't have redundancy, backup," Coulter said yesterday. "The contract does not call for redundancy in carriers . . . in the network.
          "Why that wasn't put into the network, I don't know," Coulter said. "This is a service we have to have."

          My guess is that the VA employees overseeing the bidding and proposal writeup did not understand the importance. They probably wanted somewhere to cut costs, saw the word 'redundant', and thought that would be a good place to save money. Without someone technical with enough weight to tell the higher-ups that this is a necessity, some manager will decide to kill that part of the proposal, thinking they saved the state millions. Presuming the la

      • Re:Blame Northrop? (Score:5, Insightful)

        by nine-times (778537) <nine.times@gmail.com> on Tuesday November 24, 2009 @10:48AM (#30213592) Homepage

        They're not in business to keep the state government afloat, their only purpose is to make money.

        I hate when this is offered as an excuse for shoddy work. "It's not their job to do good work. It's their job to make money." Yeah? So what. It strikes me a little like saying, "Hey, can't blame a con man for stealing your money. That's what con men do!"

        I don't know this particular situation well enough to say who is at fault and to what degree, but it's part of their business to service their customers well. It's part of every company's business to provide service to their customers in an ethical manner.

        • by idontgno (624372)

          I don't know this particular situation well enough to say who is at fault and to what degree, but it's part of their business to service their customers well. It's part of every company's business to provide service to their customers in an ethical manner.

          It's also a part of every publicly-traded company's business to provide maximum value to shareholders in an ethical manner. And that is also a legal requirement [wikipedia.org]. Failure to do so will get your board or your executives in front of a judge so fast they'll

        • by tnk1 (899206)

          I generally agree with your sentiment about businesses wanting to do the most for their customers, but it's a little different in government. The government needs to solicit bids every time it asks for something and as mountains of red tape. This tends to prevent a business from generating a good relationship with government agencies based on providing things that go above and beyond what is requested. The government can never just sort of call up the sales rep of a company that they have a good relation

    • Re: (Score:2, Insightful)

      by Publikwerks (885730)
      It's the state's fault for not putting that in the contract. I have worked for state contractors who handle IT services, and the state always had a downtime penalty written in to the contract, so it was too expensive to be down not to have a redundant system. This is probably a case of penny pushers not doing their homework, seeing that one system is cheaper than two.
    • How do you know that Northrup didn't suggest them? Or that Northrop wasn't the lowest qualified bidder by being the only one not to include backups in their bid? When you buy a computer do you expect the vendor to throw in a free backup system even though you didn't ask for it "because you should have it"?

      Sounds to me like the state got what it ordered.

    • In my experience, it is rare for a customer, even with professional IT staff, to properly specify their needs when it comes to technology. Why did Northrop, which presumably has experience in government systems, not design backups?

      Consultants never seem to get it right. Granted, the situations I'm familiar with are a fraction of a fraction of a percent of the size of this contract but at that scale the issue is that the contractors and consultants are juggling multiple clients so the needs of any one must be balanced against the needs of all of them. But what really burns me is when they can't even provide decent advice on what should be bread and butter. "We need a backup solution." See, there you go. Many solutions on the market bu

    • Re: (Score:3, Insightful)

      by Chris Mattern (191822)

      Welcome to the world of government low-bid contracts. The specification didn't call for backups, so you don't get backups, because that would've made the bid higher.

      • Welcome to the world of government low-bid contracts. The specification didn't call for backups, so you don't get backups, because that would've made the bid higher.

        It would be interesting to know who developed the specification - an intern in State IT or a professional software 'architect'.

        This is a particularly vexing problem for local government, without the resources to properly specify, but I'd hope that the Commonwealth of Virginia could do better.

      • by jcnnghm (538570)

        Welcome to the world of uninformed slashdotters. This contract was likely best value, not low-bid. Low-bid procurements are pretty rare, especially for something like this. It's not the contractors job to overbid the contract to provide services the government decided it didn't need.

    • by magarity (164372)

      From the article is seems that NG installed and ran the system for a while but only after the state people took over did it experience a lot of problems. I'm curious what is causing these outages before I'll blame either NG for not installing backups or the state for not budgeting them.

    • by Fallon (33975)
      Probably because the contract that the state gave NG didn't specify any kind of redundant networks. If a contract specifies XYZ, you have to provide XYZ, not XYZ + a redundant network. And before you start screaming about why NG should provide redundant networks anyway, contracts like that specify to some degree what money should be spent on, and you can't steal money from other parts of the contract to provide something that isn't contractually obligated.

      Oh, and the guy that was the CIO overseeing this pro
  • by master_p (608214) on Tuesday November 24, 2009 @09:24AM (#30212704)

    Have you ever seen backup systems in Star Trek, for example? you haven't. The future requires no backups.

    • Re: (Score:3, Insightful)

      by mikael_j (106439)

      Actually, there are plenty of backup systems in Star Trek. Of course, a few of them fail in every episode to avoid having every episode end with a "Yay for Starfleet engineers!" after five minutes.

      In fact, for some systems they apparently have up to four backups which all manage to fail magically at the same time *cough*transporters*cough*.

      /Mikael

      • by corsec67 (627446)

        Talking about that, is there a single instance in Star Trek where the "Manual Override" actually worked?

        And then, how is it a manual override if you just flip some other switches. The way they use "manual overrides" in Star Trek the bridge should be similar to that of the Tardis.

        • by Knara (9377)

          There was one time when Picard had to manually control thrusters to get out of an asteroid belt that had a trap in it, IIRC.

          Let us not speak of the "manual control" in "Insurrection"

    • by mcgrew (92797) *

      Actually, yes. In one episode of NG they had to purge the system of malware and indeed used their backup. The only time I ever saw a backup where I work was when it was part of my job to change the tapes.

      • by rwv (1636355)

        when it was part of my job to change the tapes.

        I had that job once. I hated that part of my job because it was repetitive and boring and the backups failed more than they should have.

        • Yeah, I spent some time doing that job as well as some time doing tech support for a major backup drive mfg... that was a horrifying experience.. I generally keep my most important files in no less than 3 locations. Not too paranoid about multiple versions. I just started using dropbox to sync my thunderbird and firefox profiles, in addition to a nas backup from my main desktop, so generally covered. Thinking about adding an additional NAS for backing up my NAS, not to mention that I have a spare drive s
    • by sjames (1099)

      The ideal backup system is invisible. You don't even know when you're using it because it took over seamlessly. An alert comes up in the noc and the primary is repaired behind the scenes.

      That doesn't work in the world we live in. Such systems cost a good bit more money and even if funded initially, will go away soon enough due to budget cuts because management says "we don't need backups because we've never had an outage" while the techs know the backups seamlessly took over 3 times in the last year.

      • by TheLink (130905)
        > The ideal backup system is invisible.

        That already exists according to the Many-Worlds-Interpretation Quantum Physicists.

        > That doesn't work in the world we live in.

        That's the big problem though - when "stuff happens" the backup is in other worlds.
    • by dkleinsc (563838)

      Sure it does. What the heck is all the talk about auxiliary and emergency power?

      Of course, the real limitation is that any redundant control systems make the operator's console twice as likely to explode.

    • Re: (Score:2, Funny)

      by czmax (939486)

      In the Star Trek future you can always route auxiliary power to the overloaded/failed device; which is usually sufficient to get to the end of the episode.

  • Easy (Score:5, Insightful)

    by Spad (470073) <<slashdot> <at> <spad.co.uk>> on Tuesday November 24, 2009 @09:25AM (#30212716) Homepage

    During the first six months of the year, state Department of Transportation workers faced 101 significant IT outages totaling 4,677 hours: an average of more than 46 hours per outage. One took 360 hours to fix.

    That's 27 weeks of downtime in the space of 26 weeks, which raises a much more important question than why there's no network redundancy and that question is: What kind of fucking morons have they got running their systems?

  • outsourcing (Score:5, Insightful)

    by Clover_Kicker (20761) <clover_kicker@yahoo.com> on Tuesday November 24, 2009 @09:26AM (#30212730)

    But I thought the magic pixie dust of free enterprise would make outsourcing something to the private sector cheaper, more efficient, and better in every possible way?

    • Re:outsourcing (Score:4, Informative)

      by idiotnot (302133) <sean@757.org> on Tuesday November 24, 2009 @09:43AM (#30212920) Homepage Journal

      Hey, it worked. Mark Warner won two-thirds of the vote in his senate run last year based on his stellar performance as governor. This was one of his big initiatives.

      (He also *fixed* the revenue sources, so that there'd never be a problem like happened with Jim Gilmore. Yet, now, Virginia is in worse shape than when he got there.)

    • Re: (Score:3, Insightful)

      by darjen (879890)

      The government is clearly involved here. So it's got nothing to do with free enterprise.

      • by circletimessquare (444983) <circletimessquare@NOSpam.gmail.com> on Tuesday November 24, 2009 @11:09AM (#30213868) Homepage Journal

        and has involved itself in the market in some way in the past

        therefore, any prudent rational criticism of the free market and how it obviously fails can be explained away with creative rationalization that its the government's fault, somehow"

        my favorite is how free market fundamentalists wish to blame the market crash of 2008 on government policies. rather than gee, i dunno, the clinton and bush administration deregulation policies? you know, deregulation: having the government less invovled int he market?

        "what? my free market bubble and pop? nah, impossible! government's fault! pffft"

        please study your banking panics of the 1800s: without regulation, free markets have innate imperfections which always result in catalcysmic failures. all you need is simple human psychology, no government need apply, to cause a market to crash. you either regulate it, leveling the playing field artificially, and therefore making it truly "free", or you leave it alone, letting it bubble and pop like mad, and allow monopolists to take advantage of natural imperfections in the market to leverage unfair behavior

        free market fundamentalism is dead. your ideology is dead. fact: you need government involvement in the market for the market to run efficiently. fact: you need government policing and regulation of the marketplace to keep it "free" and egalitarian and equal for all players

        if you don't understand these simple truths by now, or refuse to believe that despite the obvious proof, you're an idiot

        • by darjen (879890)

          the government exists and has involved itself in the market in some way in the past

          therefore, any prudent rational criticism of the free market and how it obviously fails can be explained away with creative rationalization that its the government's fault, somehow

          so the Virginia state government is not actually in charge of their own damn IT system? they were only involved in their own IT system in some way in the past? interesting theory. please tell me more.

          my favorite is how free market fundamentalists wi

          • would you like actually come out and refute my statement that you need government regulations to keep markets fair?

            or are you just raging against a truth that hurts, but that your mind accepts?

            thought so

    • by Korin43 (881732)
      Yup, that's what the free market is all about: The government paying someone to do something.
    • by jcnnghm (538570)

      Government can't properly spec a problem causing outages, and it's the fault of "free enterprise". You people sure have a vivid imagination. From the article:

      "The problem of no-redundancy . . . accounts for 90 percent of our outages," said David W. Burhop, the DMV's chief information officer.

      "The first thing I noticed was that the network that Northrop Grumman rolled out didn't have redundancy, backup," Coulter said yesterday. "The contract does not call for redundancy in carriers . . . in the network.

      Smells like government incompetence.

  • by Cornwallis (1188489) on Tuesday November 24, 2009 @09:28AM (#30212752)

    Remember how Virginia's health records were compromised earlier this year?

    http://it.slashdot.org/article.pl?sid=09/05/05/1232240 [slashdot.org]

    Sounds like systemic ineptitude which is why I'm really looking forward to more government involvement in health care!

    • by east coast (590680) on Tuesday November 24, 2009 @09:57AM (#30213076)
      That wasn't a compromise of health records. That was transparency!
    • by mcgrew (92797) * on Tuesday November 24, 2009 @10:01AM (#30213122) Homepage Journal

      Bureaucracy is bureaucracy. [itbusiness.ca] Government involvement doesn't mean ineptitude, and the free market doesn't gurantee competence. Whether private or public, ineptitude as well as competence abounds.

      • by 5KVGhost (208137)

        That's often true. But private ineptitude tends to be a self-correcting problem. Businesses that are consistently unresponsive to the market and that do really stupid things will fail. And they should fail. Better businesses spend a great deal of effort to identify their mistakes so that they won't repeat them.

        Public bureaucracies, OTOH, are essentially pure monopolies. They entrench themselves and always outlive whatever original purpose they had, they're given the power of law to enforce their decisions,

        • by Uberbah (647458)

          But private ineptitude tends to be a self-correcting problem.

          Yes, as evidenced by the fact that all the bankers and investors that drove our economy into the ground are now delivering pizza and mopping floors. Oh wait, they're getting bigger bonuses than ever.

          What you describe works nice in theory, but in practice executives will take enormous risks at the chance of enormous short term profit. Why make a million dollars a year through responsible investment if you can make 30 million dollars in two years?

      • by rwv (1636355)

        Whether private or public, ineptitude as well as competence abounds.

        It seems like in Virginia... ineptitude abounds.

        I live in Massachusetts... competence is fairly prevalent here. Going to the RMV (our version of the DMV) still sucks, but unless you're getting your license for the first time you can circumvent that hellish trip and process your registration forms through a website. :)

    • by Jawn98685 (687784)

      Remember how Virginia's health records were compromised earlier this year?

      http://it.slashdot.org/article.pl?sid=09/05/05/1232240 [slashdot.org]

      Sounds like systemic ineptitude which is why I'm really looking forward to more government involvement in health care!

      Yeah, right. Because the private sector has done such a good job of protecting our privacy, banking info, etc.
      Please..., go troll somewhere else.

      • Actually, that breach happened after NG took over the systems. The contract has been in place for 4 years.
    • by ffflala (793437)

      Sounds like systemic ineptitude which is why I'm really looking forward to having health care!

      FTFY

    • by Uberbah (647458)

      Sounds like systemic ineptitude which is why I'm really looking forward to more government involvement in health care!

      And how is your Enron stock doing these days? Managed to avoid electrocution in your bathroom from shitty contractor wiring?

  • NG, I call you out! (Score:4, Interesting)

    by Anonymous Coward on Tuesday November 24, 2009 @09:29AM (#30212756)

    Most forget that the network provided by the NG crooks is NOT part of the Comprehensive Infrastructure Agreement (CIA). It is a seperate agreement that is a fixed cost agreement under which NG was supposed to replace “like for like”. They were supposed to install an MPLS network. MPLS (Multi Protocol Label Switching) allows for the prioritization of traffic to allow Voice traffic to travel over the same circuit as the data. It also supposed to be intelligent enough to encrypt data to essentially allow a VPN to be created from point-to-point.

    None of the VPN has been done as promised, very few sites have used the VOIP option unless dictated to by VITA as part of new construction and most sites complain about network performance. Some agencies had totally redundant networks but were forced to pay more for less. 65% of VITA staff make over 90,000 a year. Again we pay more for less.

    While I am not a NG fan, interestingly enough, most state managers at Agencies will tell you that working directly with NG allows things to get done, VITA just gets in the way. VITA wants to always be the interface, Waste Fraud and Abuse to pay high salaries for mostly unqualified folks. Throw out VITA and let the agencies be treated like customers by NG.

    The IT Community Frowns Upon Your Shenanigans...

  • by Joe The Dragon (967727) on Tuesday November 24, 2009 @09:35AM (#30212838)

    Northrop Grumman outsources part of it's own IT as well and it does not own some of it's systems they rent them or at least they did 1-2 years ago.

  • by tomhath (637240) on Tuesday November 24, 2009 @10:00AM (#30213110)
    Is seems nobody RTFA (no surprise). The problem they're having is network outages at branch offices. I assume they're using DSL or such, with no way to connect if/when it goes down. Any one office probably has >99% up time, but when you have hundreds of offices and the remnants of a hurricane come through you can expect several of them to go offline, which is what's happening.
    • s/Backup/Redundancy/ and half the comments go away - vague headlines must be Slashdot's new evil plan to get page views.

    • If you finished RTFA, they didn't have these problems until it was outsourced.

      "But without backup circuits -- which VDOT had before the Northrop Grumman outsourcing -- to take up the load, the transportation agency's Hampton Roads' IT network went out of service 23 times during the event."

      So during the planning stage, someone in the gov't and NG f'ed up in not seeing that backup lines were in place, and they should stay in place. But that aside, why are the lines going out is the question? Was NG contract
    • by icepick72 (834363)

      >> with no way to connect if/when it goes down.

      Ironically the Internet was supposed to solve that problem. Maybe they simply need multiple Internet connections from different providers and across different mediums: microwave, satellite, dial-up (yuck), pigeon. Expensive.

    • by King_TJ (85913)

      And if you ask me? As much as anything, this illustrates why the nation's broadband infrastructure is sadly inadequate....

      What are you supposed to do in many of these locations as an alternate to a DSL circuit? Dial backup with 56K modem, I suppose? Oh yeah, THAT will run great with today's bandwidth-saturating apps.

      Oh, perhaps they should spend the money for satellite Internet at each location then? Big up-front equipment and setup expense for something with high latency and relatively high monthly cos

  • by zerofoo (262795) on Tuesday November 24, 2009 @10:01AM (#30213124)

    The article does not mention "backups" as in tape drives and off-site storage.

    The article does mention lack of redundancy at the network carrier level.

    My guess is that Northrop Grumman designed a network around single circuits connecting offices to data centers, and did not design the network to tolerate WAN link failures.

    A stupid oversight for sure, but nothing that can't be easily remedied by ordering redundant WAN circuits from your telco of choice. Redundant routing gear would also be smart.

    For all that are blaming government for this - they outsourced the design and implementation to a private company. That company screwed the pooch in design and implementation. Shame on both parties for not recognizing the risk of WAN failure.

    -ted

    • by raddan (519638) *
      This is also something that can be addressed in the design of the software at the site. As someone who runs a site with redundant links, I can say-- it's not for everyone. It is very expensive, and running requires a high level of competence, which also means $$$.

      OTOH, if you design your software so that your software can tolerate periodic downtime, this kind of problem can be mitigated. Instead of the knee-jerk reaction, "install redundant links!", VA should probably analyze the data on failures and
  • Epic Fail (Score:2, Interesting)

    by halfEvilTech (1171369)

    If any story deserves this tag it is this. from the article:

    "Virginia declared a state of emergency Nov. 11 in the face of record nor'easter rains and winds.

    But without backup circuits -- which VDOT had before the Northrop Grumman outsourcing -- to take up the load, the transportation agency's Hampton Roads' IT network went out of service 23 times during the event.

    "We called at 5:35 in the morning," said Gary Allen, VDOT's chief of technology, research and innovation.

    "It took VITA four hours to open the hel

    • Didn't you hear? There was a storm man... It was cold, windy and raining oceans of water. We were told to stay home. The tech was probably in his bed in the dark (no electricity) trying to ignore the ringing cell phone.
  • by Cprossu (736997) <cprossu2&gmail,com> on Tuesday November 24, 2009 @10:13AM (#30213244)

    "During the first six months of the year, state Department of Transportation workers faced 101 significant IT outages totaling 4,677 hours: an average of more than 46 hours per outage. One took 360 hours to fix."

    wait, 4,677 hours? how could that be? There were 181 days in the first 6 months of this year, that's only 4,344 hours.. there was more downtime on the system than days in it's operational life! (did someone /0 here?)

    Outsourced, no thanks... I think I'd rather dig up a Univac I to do work on, at least it would be more reliable

    • Re: (Score:2, Informative)

      by halfEvilTech (1171369)

      My guess is that would be multiple systems. They noted in TFA that they provided IT services to 1000 local governments and 85 state agencies in VA.

    • Re: (Score:3, Insightful)

      by Tino (1418)

      4,677 hours of failure in 4,344 hours of time means that at any given time, an average of 1.07 locations were offline.

      There are 131 DMV offices in Virignia; I don't know how many other Department of Transportation locations are included in the same bucket. If we assume that it's *only* the 131 DMV offices, 1.07 failures at any given time means that the system means that 130.3 locations are working, meaning that this statewide patchwork of network connections is 99.45% reliable.

      If your 'redundant' connectio

  • by volxdragon (1297215) on Tuesday November 24, 2009 @10:18AM (#30213286)

    I *knew* there had to be some other reason why they closed half the interstate rest stops in VA, this is obviously where the money was (mis)spent...

  • by tinkertim (918832) on Tuesday November 24, 2009 @10:23AM (#30213348) Homepage

    From TFA:

    During the first six months of the year, state Department of Transportation workers faced 101 significant IT outages totaling 4,677 hours: an average of more than 46 hours per outage. One took 360 hours to fix.

    Suddenly, I don't feel so bad for that 2 1/2 hour glitch last week :)

  • Northrop Grumman's core business is making airplanes (at least it was). So what we have here is a non-core business effort on behalf of a state government contract. I'll bet it was staffed by the B team at Northrup Grumman because real IT hot shots just are not motivated to get out of bed in the morning to chase state government contracts. On top of this staffing issues, I'm sure the government had lots of non-standard 'requirements' from insecure bureaucrats that need to justify their jobs. This is a le
  • Living in the DC area i see all sorts of crap going on with contractors. The amount of money the fed wastes on contractors should cause us to rise up and slay them. But so many profit (my GF and i included) that it seems unlikely that it will change soon. The CIA for instance has crippled itself by using so many contractors, people who SHOULD be on the payroll. Instead they line the pockets of executives and share holders, shitting away millions upon millions in overhead costs. This means much of the ta

  • Looks like Big Government screwed up again, should have gone with the private sector!

Whoever dies with the most toys wins.

Working...