Forgot your password?
typodupeerror
This discussion has been archived. No new comments can be posted.

Passport Database Outage Leaves Thousands Stranded

Comments Filter:
  • by dave562 (969951) on Thursday July 31, 2014 @06:57PM (#47578095) Journal

    Rollback plan? What is that?

    • Yeah I was going to say the same thing. You NEVER make a change that you don't have a way to backout to the previous operational state.
      • by roc97007 (608802) on Thursday July 31, 2014 @07:15PM (#47578187) Journal

        It's the wave of the future. A typical contract with offshore IT is for "current minus one", which means that each new firmware, OS or driver release causes a flurry of "maintenance" by remote "admins" who follow written procedures to update the systems with no real understanding of what they're doing, in what order they should do it, or what to do if something goes wrong. A typical list of systems to update may randomly contain a haphazard collection of prod and development machines, and may include some but not all members of a cluster. Systems are patched in Asset Management order, with no thought to rolling through dev and QA first before doing prod.

        The backout plan is to engage the vendor.

        Our outsourced IT bricks a few servers a year. We try to take it in stride. We've argued hysterically that if they really have to do firmware updates, to at least do dev servers first for God's Sake. They seem to not understand this.

        So yeah, I could definitely see this happening. We will be seeing more of same. You get what you pay for.

      • by roc97007 (608802)

        ...in case my other article did not make it clear, we always ask if they have a backout plan, and they always say they do.

        ...and then, when the system does not reboot after an update, we find out that the backout plan is to call Dell and say "the system, it is not being working. What are we to be doing?"

        ...and we pay money for this.

        • by Anonymous Coward on Thursday July 31, 2014 @07:33PM (#47578309)

          Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no". The answer is always "yes" or some other affirmative that makes you think they have it under control and can do the work. When the fact is, the work they just said "yes" to, they don't actually have a clue how to perform it, so they learn as they go, on your production servers. They don't know what development / test environments are.

          • by ShanghaiBill (739463) on Thursday July 31, 2014 @08:38PM (#47578609)

            Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".

            It takes two to fail to communicate. You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.

            • Re: (Score:2, Insightful)

              by Anonymous Coward

              You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.

              Do I care?

              • Re: (Score:2, Insightful)

                by Anonymous Coward

                You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.

                Do I care?

                If affects your ability to do your job, you should.

                • by ruir (2709173) on Friday August 01, 2014 @04:00AM (#47579851) Homepage
                  Well, it also affects my ability to do the job people lying to me or choosing to reply with half truths to save face. My culture considers that extremely rude too. The rules of engagement have to change in a multicultural world, and if I am the customer, their obligation to bend somewhat their culture is a ball on their side. Or I may take my business and wallet elsewhere.
                  • by ihtoit (3393327)

                    This. So THIS! I cannot and will not do business with liars. If I can't get a straight answer to a straightforward question, I'll go elsewhere.
                      It also helps my clients if they're up front, honest and straightforward with me. In ascertaining their requirements, I ask exacting questions, if they think I'm being anal, well, they just found out why I'm so fucking expensive. It's because the job isn't done until it's right.

                    • by ruir (2709173)
                      I often have this problem with suppliers and in the past even had it with subordinates. Nowadays no suppliers at all get root, and I have configurations in place where all their commands are logged to the system logs.
            • by Loki_1929 (550940)

              Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".

              It takes two to fail to communicate. You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.

              So they lie because their culture tells them to and it's my fault for not identifying that they're lying and taking careful steps to help them not lie?

              Sorry, but that's absurd. If one's culture does not allow one to perform one's job correctly, one needs to either find a new culture or find a new job.

              • The problem is that that's often not how they think. In some cultures you only ask a yes or no question in order to hear a yes answer, no matter what. You think you are enquiring for information. The poor Elbonian you are talking to may be totally stressed out because he thinks you are telling them that you are very angry about him because you lost face before your superior due to his poor performance. And that, even though nobody knows or can know how to do things better, you now have to report back to tha

            • by roc97007 (608802)

              Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".

              It takes two to fail to communicate. You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.

              Um, ok.... this is IT. Sometimes yes or no questions are required. If the outsourced IT company can't culturally or by whatever reason answer a yes or no question, should we be doing business with them?

              I mean, if this is a cultural problem throughout India (and I don't really believe that's true) how do they have, for instance, a space program? How can anything that requires precise communication be made to work?

              Is it possible that the real reason why they always answer "yes" even when they clearly don't

            • by PJ6 (1151747)

              Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".

              It takes two to fail to communicate. You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.

              Really? Could a question like, "Is it raining?", which presumably would be answered by a yes or a no, really be considered rude?

              I'm at a total loss to see how any interpretation of your assertion could be true.

              Could you give us a simple and clear example, and then provide another in the context of a work environment, where management is expected and acceptable?

          • by khchung (462899) on Thursday July 31, 2014 @11:13PM (#47579129) Journal

            Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".

            On the other hand, I have encountered plenty of managers who literally do not know how to take "no" as an answer.

            Takes two to make a pair.

          • Outsourcing is bad, but as an Indian who renewed his passport recently, I have to say the process was smooth.
            The document verification/IT/software/hardware part of the Indian passport application/renewal operation is now handled by TCS. You make an appointment online - for both fast track and slow track - and arrive at the local passport center with your documents. The TCS grunts allot a token, make sure documents are in order - if they are not you are sent back to get them corrected, take your fingerprin
          • by Jesrad (716567) on Friday August 01, 2014 @05:05AM (#47579985) Journal

            Sounds like your IT has been outsourced to India

            Not necessarily. I've seen this exact kind of madness happen just as easily with locals, here in France. Like that time the local, on-site support team from our vendor rebooted the production server instead of the test platform, because woops wrong terminal window in the foreground.

            Or when they covertly rolled out a "shame-bug fix" remotely on the production platform during a week-end night, again instead of targetting the test platform, then noticed their mistake, and wiped-out months of production data by reverting to a long-expired backup.

            Or when the local datacenter people managed to botch our fully-automatized install+deploy+configure solution by messing up on the one thing they had to do right - that is, upload it and launch it on the correct machine of the cluster.

            Don't think hiring local people for more money protects you from such cringe-worthy nonsense. The moment you outsource anything, and I do mean *anything*, no matter how far and how expensive and what nationality: if you base your expectations on anything but an actual track-record of reliability and dependability, you're exposing yourself to long hours of hair-pulling and yelling into phones.

        • ...in case my other article did not make it clear, we always ask if they have a backout plan, and they always say they do.

          I used to deal with a lot of Indian outsourced IT groups, and the only way to handle this is to either follow up the "Yes, we have a backout plan" response with "Tell me what your backout plan is" or just to skip straight to that without bothering to ask the "Do you have a plan?" question.
          Things still got screwed up, but after the first occurrence we completely cut their access to the servers and re-enabled them on demand, so we forced their people to update a specific server first to show that they could d

          • by roc97007 (608802)

            I think cutting their access and adding it back on a case by case basis is a brilliant idea. But I fear we as a company have lost control of the environment. We don't have root anymore, except for the service accounts, and sudo lists that they've forgotten to, or don't know how to, plug. And we can't seem to convince upper management that even the developers who used to be admins could do a better job than the people we have now.

      • by gstoddart (321705)

        Yeah I was going to say the same thing. You NEVER make a change that you don't have a way to backout to the previous operational state.

        And, really, if you have something which Absolutely Has To Be There ... you make damned sure you have an environment you apply the changes to first. So that you can apply the changes and at least try to make sure stuff don't break without messing up the real one.

        This is basic change management.

        (And, yes, I am saying this without any context for this outage -- but, really, i

    • by ruir (2709173)
      Forget the obvious rollback plans? At the very least working without computers, no? Or having alternative system for something so crucial...If computers are only there to complicate, than we are better off without them. The level of incompetence and lack of planning is strong in several levels.
      • by dave562 (969951)

        You bring up a good point. Given the extent out of the impact of the change, they probably should have just declared a disaster and gone with Plan B. Yet, given that they blew a system change and did not have a rollback plan, I am fairly confident that any sort of DR strategy is equally broken and worthless.

        Situations like this always put a smile on my face, because I know that my job is secure. If an organization as large as the United States government cannot get these basics right, but I can... I know

        • by ruir (2709173)
          This kind of situation is unthinkable in 2014...haven't they heard about virtualisation technologies and clones before major changes (as at least a very minimum baseline?) How about pre-production system to TEST changes? Change management process? Backups? Alternative systems for disasters replicated from last week systems? If I had to guess, I would say they where hacked and could not count on backups, but then we are dealing with the government and DHS, and they are often known for not being that competen
          • by dave562 (969951) on Thursday July 31, 2014 @08:41PM (#47578621) Journal

            As much as I am not a fan of government regulation, my professional experience has shown me that the only time people get IT anywhere close to right is when there is a risk of financial penalty involved in getting it wrong. Regulation seems to be the only solution to people working for peanuts. The people who work for peanuts make mistakes. If those mistakes cost the company more than the company saves by hiring those people, they will not hire those people.

            Out of all of the industries that I have worked with, the financial services industries seem to be the most together. They are not perfect, but the penalties associated with losing customer data makes them more careful.

      • by dbIII (701233)
        There are sometimes very "political" problems that prevent rollback because for instance that means buying more time on the licences from an earlier vendor when the new vendor is heavily embedded with the management that are driving the change. Such a problem in my state resulted in such an enormous fuckup in Hospital payroll systems with no rollback that there was real political fallout - after the bill went beyond 500 million the government lost office to be replaced by a bunch of baby fascists led by a
        • by ruir (2709173)
          Very interesting insight sir, we often are very aware of the political problems in our backyard, but forget others have such similar problems.
          • Ultimately my example was about: "level of incompetence and lack of planning is strong in several levels", as you suggested but it was driven that way by the new vendor having far too much control over the situation and no risk to bear in the event of failure.
            The government took them to court twice (outgoing and incoming - Queensland, Australia) and could not scratch that vendor (IBM) for any of the $500 million+ in estimated extra costs.
    • by Ol Olsoc (1175323)

      Rollback plan? What is that?

      Umm - have you tried rebooting the computer sir. Try that first.

  • They are hard drive experts!
  • Still, bet Sysadmin's the highest ranking head that'll roll.
    • by ruir (2709173)
      Will they? Apparently they do not have sysadmins and are just a bunch of system operators, if that.
  • by Loopy (41728) on Thursday July 31, 2014 @07:11PM (#47578181) Journal

    Sic the healthcare.gov guys on it. I'm sure it'll be right as rain in no time.

  • by Anonymous Coward on Thursday July 31, 2014 @07:19PM (#47578219)

    From their Q&A:

    Q: Why wasn’t there a back-up server?
    Back-up capability and redundancy are built into the system. The upgrade affected our current processing capability, in part because it interfered with the smooth interoperability of redundant nodes.

    We don't need backups, the data is replicated, we're cool.

  • We call this being over improved. So much for testing.
  • Ask the NSA (Score:5, Funny)

    by Daemonik (171801) on Thursday July 31, 2014 @07:33PM (#47578303) Homepage
    I'm sure they have full copies of all the data already.
  • by ruir (2709173) on Thursday July 31, 2014 @07:36PM (#47578319) Homepage
    That these breakdowns are lame excuses. If computers fails, have people forgot how to do the same process manually? It is better to halt all the flights than letting people through and risk "terrorists" flying? Are we that terrified?
    • by Kittenman (971447)

      That these breakdowns are lame excuses. If computers fails, have people forgot how to do the same process manually? It is better to halt all the flights than letting people through and risk "terrorists" flying? Are we that terrified?

      You could just ask the questions that used to get asked back in the '50s. "Do you intend to bring down or otherwise defame the US government?"

      Really. A UK humourist (Frank Muir?) wrote "Sole purpose of visit" on the form.

    • by dbIII (701233)

      If computers fails, have people forgot how to do the same process manually?

      Yes.
      As an example. I've been rushed to a steel mill rolling line with a pocket calculator because the operators were not taught how to divide the number on the dial of the test machine by the cross sectional area of the rod that they had measured the diameter of - they were just told to manually enter those two numbers into the computer. By knowing how to calculate the area of a circle I was saving downtime of hundreds of thousand

  • by Anonymous Coward

    The whole US customs and immigration system is massively dysfunctional. Last year I flew into Minneapolis from Asia. I'd been traveling for twenty hours straight and then I got to stand in line for a full hour waiting for an immigration agent to spend ten seconds looking at my passport photo to make sure it matched my face. Even the third world airports I've been through aren't that bad. There were even empty stations without agents. How much would it have cost to add a few more agents - $100? At the time t

    • by CRCulver (715279)

      At the time they were doing this ridiculous upgrade to the airport that must have cost millions - they were setting up all these silly little tables with ipads in the waiting areas. But somehow they couldn't manage to have enough immigration agents.

      Airport infrastructure is typically managed by a consortium with private or state-level involvement. US customs officials are federal employees. The two parts of the airport have entirely different funding sources.

    • by PPH (736903)

      They have most of the lines closed so they can back people up and watch who is starting to break out in a cold sweat.

      Which is why most of the bad guys fly first/business class. Faster through the lines. Less chance of being observed.

    • Customs and immigration agents are Federal employees, and airport upgrades are state and private level. I don't know if an airport can simply pay Customs to supply more agents.

      In any case, having another station open would cost a whole lot more than $100. There isn't a 24/7 pool of available customs agents who can simply show up for $50/hour, two hours minimum, when there happens to be a lot of people coming through. To man a station 24/7/365 (shouldn't that be 24/7/52.14?) takes a minimum of five peo

  • by Anonymous Coward

    I think I found the problem, from the Department of State's own website:

    "The Department of State is working with Oracle and Microsoft to implement system changes aimed at optimizing performance and addressing ongoing performance issues."

    They're running Oracle on Windows.

  • I have arrived at the point where any crashes experienced by whatever State Department of whatever so called and self proclaimed Democratic Country (traitor mark here) are welcomed by me with the utmost glee. The more disruption, the more chances for a turnaround.

  • Large Databases? (Score:5, Insightful)

    by TechyImmigrant (175943) on Thursday July 31, 2014 @10:46PM (#47579055) Journal

    The article tries to wow us with the hugeness of the database, like this is a reason for the issues.

    Yet the numbers quoted are not that big. Any modern PC isn't going to get too upset handling 75 million things. A real data center is going to sit there wondering what to do with the remaining 500TB of storage.

    I don't doubt that there is some horrible flaw in the way the system was conceived that rendered it fragile, but whatever it is, it's nothing to do with the enormity of the problem, because it isn't very enormous.

    • by ihtoit (3393327)

      75 million page-sized data objects (let's call it "text") would fit comfortably on my laptop hard drive*.

      *2x2TB, I feel like I'm flapping my cock out when I'm sitting in a coffee bar playing world of tanks on it.

"Mach was the greatest intellectual fraud in the last ten years." "What about X?" "I said `intellectual'." ;login, 9/1990

Working...