Passport Database Outage Leaves Thousands Stranded 162
linuxwrangler (582055) writes Job interviews missed, work and wedding plans disrupted, children unable to fly home with their adoptive parents. All this disruption is due to a outage involving the passport and visa processing database at the U.S. State Department. The problems have been ongoing since July 19 and the best estimate for repair is "soon."
The system "crashed shortly after maintenance."
Change management fail (Score:5, Insightful)
Rollback plan? What is that?
Re: (Score:2)
Re:Change management fail (Score:5, Informative)
It's the wave of the future. A typical contract with offshore IT is for "current minus one", which means that each new firmware, OS or driver release causes a flurry of "maintenance" by remote "admins" who follow written procedures to update the systems with no real understanding of what they're doing, in what order they should do it, or what to do if something goes wrong. A typical list of systems to update may randomly contain a haphazard collection of prod and development machines, and may include some but not all members of a cluster. Systems are patched in Asset Management order, with no thought to rolling through dev and QA first before doing prod.
The backout plan is to engage the vendor.
Our outsourced IT bricks a few servers a year. We try to take it in stride. We've argued hysterically that if they really have to do firmware updates, to at least do dev servers first for God's Sake. They seem to not understand this.
So yeah, I could definitely see this happening. We will be seeing more of same. You get what you pay for.
Re: (Score:2)
...The backout plan is to engage the vendor.
Preferably with shells from an Iowa-class battleship?
Re:Change management fail (Score:5, Funny)
Well there's your problem! God has no part in an IT management plan.
Yeah, the other guy has it well in hand.
Re:Change management fail (Score:4, Funny)
I sometimes think that if I accidentally entered a church with an IT management plan in a back pocket, my pants would burst into flames.
Re: (Score:3, Insightful)
Re: (Score:2)
Sorry, I guess I deserved that. I meant, of course, that the plan would burst into flames.
Re: (Score:2)
Re:Change management fail (Score:4, Insightful)
Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no". The answer is always "yes" or some other affirmative that makes you think they have it under control and can do the work. When the fact is, the work they just said "yes" to, they don't actually have a clue how to perform it, so they learn as they go, on your production servers. They don't know what development / test environments are.
Re:Change management fail (Score:4, Informative)
Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".
It takes two to fail to communicate. You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.
Re: (Score:2, Insightful)
You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.
Do I care?
Re: (Score:2, Insightful)
You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.
Do I care?
If affects your ability to do your job, you should.
Re:Change management fail (Score:5, Insightful)
Re: (Score:2)
This. So THIS! I cannot and will not do business with liars. If I can't get a straight answer to a straightforward question, I'll go elsewhere.
It also helps my clients if they're up front, honest and straightforward with me. In ascertaining their requirements, I ask exacting questions, if they think I'm being anal, well, they just found out why I'm so fucking expensive. It's because the job isn't done until it's right.
Re: (Score:2)
Re: (Score:2)
Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".
It takes two to fail to communicate. You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.
So they lie because their culture tells them to and it's my fault for not identifying that they're lying and taking careful steps to help them not lie?
Sorry, but that's absurd. If one's culture does not allow one to perform one's job correctly, one needs to either find a new culture or find a new job.
Re: (Score:2)
The problem is that that's often not how they think. In some cultures you only ask a yes or no question in order to hear a yes answer, no matter what. You think you are enquiring for information. The poor Elbonian you are talking to may be totally stressed out because he thinks you are telling them that you are very angry about him because you lost face before your superior due to his poor performance. And that, even though nobody knows or can know how to do things better, you now have to report back to tha
Re: (Score:2)
Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".
It takes two to fail to communicate. You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.
Um, ok.... this is IT. Sometimes yes or no questions are required. If the outsourced IT company can't culturally or by whatever reason answer a yes or no question, should we be doing business with them?
I mean, if this is a cultural problem throughout India (and I don't really believe that's true) how do they have, for instance, a space program? How can anything that requires precise communication be made to work?
Is it possible that the real reason why they always answer "yes" even when they clearly don't
Re: (Score:2)
Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".
It takes two to fail to communicate. You should not be asking questions that require a direct "yes or no" answer. In many cultures, that is considered rude.
Really? Could a question like, "Is it raining?", which presumably would be answered by a yes or a no, really be considered rude?
I'm at a total loss to see how any interpretation of your assertion could be true.
Could you give us a simple and clear example, and then provide another in the context of a work environment, where management is expected and acceptable?
Re:Change management fail (Score:5, Insightful)
Sorry, what part of paying you to do a job requires me to give a shit about whether or not your failed third-world culture doesn't like answering direct fucking questions?
The part about you paying them far less than you would pay someone culturally compatible. If you want to pay peanuts, you need to deal with the cultural consequences. I have dealt with Indians for years, and have learned how to ask questions so that I get the answer I am looking for. It is not that hard.
Re: (Score:3)
Re: (Score:2)
The question is that exactly because of that culture, they seem able to complicate the most simple of the problems looking through the eyes of our culture. It also does no help that they say they know everything, their CV looks like far better than a Linus Torvalds, lots of IT certifications, but when your start asking them the simplest of the questions, they fall apart.
Given that there's only about a billion of them, I would say it is safe to assume that they all fall apart and that there isn't a single Indian how actually knows anything.
It's almost a miracle that they've managed to come this far.
That's a bit disingenuous. We're talking specifically about the people hired by companies in the business of outsourcing IT. That has to be a pretty small subset of a billion.
And as I said in another article, it's not that way because some entire race of people are stupid, but because of the way the business model is set up. In order to offer IT labor at an attractive price while including overhead and profit margin, it becomes necessary to buy the cheapest labor possible. This tends to present to the c
Re: (Score:2)
there's a viable industry around inflated resumes. I've seen some real gems.
Re: (Score:3)
Re: (Score:2)
Re: (Score:2)
I don't think it's so hard. Extreme example:
"Some of our computers are more important for our business than the others. For example, every hour that kremvax is not running costs us roughly half a million dollars because people worldwide then buy somewhere else.
It often happens that new firmware from the vendor does not work properly. So suppose you make a firmware update on kremvax, reboot it, and it does not start. What will you do in that case?" ...
"That's probably the best thing you can do in this situat
Re: (Score:2)
I don't think it's so hard. Extreme example:
"Some of our computers are more important for our business than the others. For example, every hour that kremvax is not running costs us roughly half a million dollars because people worldwide then buy somewhere else.
It often happens that new firmware from the vendor does not work properly. So suppose you make a firmware update on kremvax, reboot it, and it does not start. What will you do in that case?" ...
"That's probably the best thing you can do in this situation, though I would ask you to also ... so that ... . Unfortunately, if this happens, it can take days to resolve the situation. Two days at half a million dollars each, that would make, um, ... How much is that, help me out please..." ...
"Yes, exactly, 24 million. That's why our last system administrator was always very scared. He wouldn't have lost his job if this happened once, but some people would have been very angry. So he was always reading those books about anticipatory system administration [I just made this term up; use a proper, searchable technical term if there is one] to minimize the danger. He always came up with new tricks. For example, there is this server called deepthought that is not doing anything really important most of the time. We call this a development server. Now when he wanted to do a firmwareupdate, what do you think: which of the servers did he do first, kremvax or deepthought?"
And you realize, by doing this you are training up a very very junior person, because experienced sysadmins should already know all of this.
And this is the point. We are told that we are paying for experienced sysadmins. What we get are people who need to be taught that you do a firmware upgrade on a test system first before you do it in production.
Incidentally, once you get that admin trained up and things running smoothly, he will find a job elsewhere for more money and you will have to start over. Bec
Re: (Score:2)
I have dealt with Indians for years, and have learned how to ask questions so that I get the answer I am looking for. It is not that hard.
Paleface no ask question with forked tongue, paleface get correct answer.
Re: (Score:2)
That must have been the most self- and Western-centric comment I've read in a while. It wouldn't hurt to realize that next to the cultural sensitivity towards direct questions,
I't also the silliest fucking thing I've ever heard. If I have the director and a couple higly placed people breathing down my neck at a phenomenal burn rate, I gotta say that if the dude from India cannot answer simple yes or no questions, The's the last of their computers that will ever enter the shipping dock.
They're giving support to Americans. This booboo feelings sensitivity BS is just one more reason the offshoring doesn't work. If I'm insulting the poor guy, I wont ever bother him or his company
Re: (Score:2)
Re: (Score:2)
Sounds like your feelings are hurt.
What an odd reaction. I did what I did and was paid very well for it because I got things done. If the help doesn't perform, you simply change the help.
Re: (Score:2)
Re:Change management fail (Score:5, Interesting)
simple yes or no questions
It is only simple because you speak English. You need to widen your cultural perspectives. In other languages, and other cultures, it is not so simple. For instance, Chinese does not even have the words "yes" and "no". If you ask a Chinese speaker if they have a pen, they will answer "have" or "not have". If you ask them if they are going to lunch, they will answer "going" or "not going". There is no such thing as a "yes or no question" in Chinese, and culturally, Chinese are much more direct than Indians or Japanese.
Re: (Score:2)
simple yes or no questions
It is only simple because you speak English. You need to widen your cultural perspectives. In other languages, and other cultures, it is not so simple. For instance, Chinese does not even have the words "yes" and "no".
I worked with a Gentleman from China. He said yes and no it with the ugliest of us Americans he worked with.
And hey, if I was working in China, dealing with culture, and and saying yes or no was offensive - then I would do as they do.
But I'm here in America, where we do talk that way. If I have to learn the social quirks of every country I'm dealing with, along with not being able to understand the "help" half of the time, it's not going to happen. The source of help is a big part of the buying decisi
Re: (Score:2)
Yes=en. Its a short syllable pronounced from the throat with the mouth closed (somewhat grunt-like but not so inelegant). This is effective for most interrogatives.
No is roughly "bu" (boo). This is more a negative so it is often combine with other words, but "bu xie" (Boo shie / no thank you) is generally effective, but in the case of the "do you have" question a straight "boo" wouldn't be correct and the "I don't have" variety is useful. Conversely, "do you want [my pen/to go for lunch]" is generally answ
Re: (Score:2)
simple yes or no questions
It is only simple because you speak English. You need to widen your cultural perspectives. In other languages, and other cultures, it is not so simple. For instance, Chinese does not even have the words "yes" and "no". If you ask a Chinese speaker if they have a pen, they will answer "have" or "not have". If you ask them if they are going to lunch, they will answer "going" or "not going". There is no such thing as a "yes or no question" in Chinese, and culturally, Chinese are much more direct than Indians or Japanese.
I think you're playing with words. The original stipulation, to which many of us are objecting, is the inability to answer a direct question in a direct manner. If you ask a Chinese speaker if he has a backout plan, and he answers "have" and actually means it, and isn't just faking it because he doesn't understand the question then he has answered the question. Language quirks are irrelevant.
Re: (Score:2)
I do find it interesting how the onus is on the native to adapt to the foreigner ONLY in your outsourcing situation.
Or would you likewise expect Chinese natives buying services from an American company to adapt?
Re: (Score:2)
At least he not a coward and is using his alias. People may fall in love with a place, and at least he is trying to improve his life abroad than being exploited back home...doing more than you probably.
Why would he be exploited back home?
This America you hate - I do not think it is as you think it is.
People think we're all kooks and nuts because we allow the kooks and nuts to have their say.
Re: (Score:2)
Re: (Score:2)
And that might just make both of us happy.
Because at least you don't have a fatso at home.
Re: (Score:2)
Re:Change management fail (Score:5, Insightful)
Sounds like your IT has been outsourced to India, who as a culture, literally does not know how to say "no".
On the other hand, I have encountered plenty of managers who literally do not know how to take "no" as an answer.
Takes two to make a pair.
Re: (Score:3)
The document verification/IT/software/hardware part of the Indian passport application/renewal operation is now handled by TCS. You make an appointment online - for both fast track and slow track - and arrive at the local passport center with your documents. The TCS grunts allot a token, make sure documents are in order - if they are not you are sent back to get them corrected, take your fingerprin
Re:Change management fail (Score:4, Insightful)
Not necessarily. I've seen this exact kind of madness happen just as easily with locals, here in France. Like that time the local, on-site support team from our vendor rebooted the production server instead of the test platform, because woops wrong terminal window in the foreground.
Or when they covertly rolled out a "shame-bug fix" remotely on the production platform during a week-end night, again instead of targetting the test platform, then noticed their mistake, and wiped-out months of production data by reverting to a long-expired backup.
Or when the local datacenter people managed to botch our fully-automatized install+deploy+configure solution by messing up on the one thing they had to do right - that is, upload it and launch it on the correct machine of the cluster.
Don't think hiring local people for more money protects you from such cringe-worthy nonsense. The moment you outsource anything, and I do mean *anything*, no matter how far and how expensive and what nationality: if you base your expectations on anything but an actual track-record of reliability and dependability, you're exposing yourself to long hours of hair-pulling and yelling into phones.
Re: (Score:2)
I used to deal with a lot of Indian outsourced IT groups, and the only way to handle this is to either follow up the "Yes, we have a backout plan" response with "Tell me what your backout plan is" or just to skip straight to that without bothering to ask the "Do you have a plan?" question.
Things still got screwed up, but after the first occurrence we completely cut their access to the servers and re-enabled them on demand, so we forced their people to update a specific server first to show that they could d
Re: (Score:2)
I think cutting their access and adding it back on a case by case basis is a brilliant idea. But I fear we as a company have lost control of the environment. We don't have root anymore, except for the service accounts, and sudo lists that they've forgotten to, or don't know how to, plug. And we can't seem to convince upper management that even the developers who used to be admins could do a better job than the people we have now.
Re:Change management fail (Score:4, Informative)
Oh man, don't get me started. It's not even clear that one would need to pay more -- we have not saved money so far by outsourcing, although the outsource company keeps telling us that savings are just around the corner. The first year, the excuse was that there is always startup issues, the second year, the excuse was that the outgoing employees did not document their jobs well enough, (probably true -- who would?) the third year the excuse was that the scope was bigger than we said it was. And so forth. Each year a new excuse and each year the total cost is more than what we were paying when we had our own IT department.
So yeah, insourcing, or at least selective insourcing, (let them keep doing what they do well, if anything) makes tremendous sense to me.
But I don't make the decisions.
And even where upper management has considered terminating our outsourcing contracts, it's only to give the contract to a different outsourcing company, which only means we're now calling a building across the street from the original building in Hyderabad. Who knows, we might even be dealing with some of the same people.
Re: (Score:2, Insightful)
Re: (Score:3)
Exactly. Consider the business model -- A company tells you that you can outsource your IT department, buy it as a service, and pay the outsourcing company's overhead and significant profit margin, *and* save money. And the only way this could possibly work is if the outsourcing company goes to the LCC (least cost country) and hires the cheapest labor possible. This is justified in that all that IT stuff, it's all just following procedures, and anyone can do that.
And of course, this is a blatant falsehoo
Re: (Score:2)
Re: (Score:2)
I often can not get the mentality of wanting to have good and competent resources for peanuts. People work for a living, if customers pay, at least pay too to the people that helps running your business along. And you have summed quite nicely why I avoid working for shops whose core business is not IT, IT are viewed like some strange pumbling guys that just suck money (i.e a center cost). And then, they probably pay more to the proper plumbers...
Don't worry, the cloud's basic purpose is to get rid of those IT guys.
Re: (Score:2)
Re: (Score:2)
Sure, the cloud runs itself magically, and you call Merlin when you have problems...That should be the most stupid comment I have read lately. The cloud is only outsourcing, you know? It is like you saying you dont need more mechanics because you go to the garage and leave there your car. ;-P
You don't think the suits are salivating over the possibilitiy of the people they can get rid of once they send as much work as possible to "the cloud"? There is a repeating cycle in the business world. They start out with in-house. Then they want to cut costs, so they outsource, then outsourceing doesn't work very well because of either costing more than it was supposed to, or it doesn't work like it was supposed to, so they start hiring in house again(and remember, you are just another customer to your
Re:Change management fail (Score:4, Interesting)
Re: (Score:2)
Not to put too fine a point on it, but I've heard that sales pitch before. I agree, HP has some truly brilliant admins, as evidenced by the "demo". The day to day reality is very much different.
Re: (Score:2)
And, really, if you have something which Absolutely Has To Be There ... you make damned sure you have an environment you apply the changes to first. So that you can apply the changes and at least try to make sure stuff don't break without messing up the real one.
This is basic change management.
(And, yes, I am saying this without any context for this outage -- but, really, i
Re: (Score:3)
Re: (Score:3)
Re: (Score:2)
DHS doesn't control the database, they outsourced it to Serco who outsourced it to G4S who outsourced it to a call centre in Bangalore who outsourced it to IBM.
Re: (Score:2)
Re: (Score:2)
You bring up a good point. Given the extent out of the impact of the change, they probably should have just declared a disaster and gone with Plan B. Yet, given that they blew a system change and did not have a rollback plan, I am fairly confident that any sort of DR strategy is equally broken and worthless.
Situations like this always put a smile on my face, because I know that my job is secure. If an organization as large as the United States government cannot get these basics right, but I can... I know
Re: (Score:2)
Re:Change management fail (Score:4, Interesting)
As much as I am not a fan of government regulation, my professional experience has shown me that the only time people get IT anywhere close to right is when there is a risk of financial penalty involved in getting it wrong. Regulation seems to be the only solution to people working for peanuts. The people who work for peanuts make mistakes. If those mistakes cost the company more than the company saves by hiring those people, they will not hire those people.
Out of all of the industries that I have worked with, the financial services industries seem to be the most together. They are not perfect, but the penalties associated with losing customer data makes them more careful.
Re: (Score:2)
Re: (Score:2)
Yes it's an example of your point (Score:2)
The government took them to court twice (outgoing and incoming - Queensland, Australia) and could not scratch that vendor (IBM) for any of the $500 million+ in estimated extra costs.
Re: (Score:2)
Rollback plan? What is that?
Umm - have you tried rebooting the computer sir. Try that first.
Re: (Score:2)
Even better, if the circumstances require significant DB changes, you back up the DB (you are backing up the DB, right?), start br
Re: (Score:2)
Re: (Score:2)
From my experience with developers, developers as Ops are a misshapen twisted wandering shambling nightmare.
Re: (Score:2)
Re: (Score:2)
Did you mean NetOps? DevOps refers to a development paradigm. If your development paradigm risks actual user-impacting down-time, you need firing ASAP.
Assuming you meant NetOps, can they live with provisioning me at least four (dev, test, UAT, and training) clones of the entire production environment? No? well then, they can make their case to the CTO whether inconveniencing the
Maybe the IRS can help (Score:2)
Damn (Score:2)
Re: (Score:2)
The solution (Score:5, Funny)
Sic the healthcare.gov guys on it. I'm sure it'll be right as rain in no time.
Replication != Backup (Score:5, Interesting)
From their Q&A:
Q: Why wasn’t there a back-up server?
Back-up capability and redundancy are built into the system. The upgrade affected our current processing capability, in part because it interfered with the smooth interoperability of redundant nodes.
We don't need backups, the data is replicated, we're cool.
Re: (Score:2)
Re: (Score:2)
Re:Replication != Backup (Score:5, Funny)
http://xkcd.com/327/ [xkcd.com]
I wonder if little bobby tables applied for a passport while this was going on?
Over Improved (Score:2)
Ask the NSA (Score:5, Funny)
I wonder (Score:3)
Re: (Score:3)
That these breakdowns are lame excuses. If computers fails, have people forgot how to do the same process manually? It is better to halt all the flights than letting people through and risk "terrorists" flying? Are we that terrified?
You could just ask the questions that used to get asked back in the '50s. "Do you intend to bring down or otherwise defame the US government?"
Really. A UK humourist (Frank Muir?) wrote "Sole purpose of visit" on the form.
Re: (Score:2)
Yes.
As an example. I've been rushed to a steel mill rolling line with a pocket calculator because the operators were not taught how to divide the number on the dial of the test machine by the cross sectional area of the rod that they had measured the diameter of - they were just told to manually enter those two numbers into the computer. By knowing how to calculate the area of a circle I was saving downtime of hundreds of thousand
Re: (Score:2)
There's process and then there's information. If a till at the supermarket breaks, once can (theoretically) still process cash purchases by hand, but how about Debit/Credit?
Yes, just use any one of the other 68 tills.
As per the article, it addresses why it was not possible to implement some manual workaround : "We cannot 'handwrite' visas because security measures prevent consular officers from printing a visa unless it is approved through our database system. Until the system is brought back to full capacity and we are able to work through the backlog, service to our customers will be below normal."
What the hell did they do in the 1950s??
Not just the passports (Score:2, Insightful)
The whole US customs and immigration system is massively dysfunctional. Last year I flew into Minneapolis from Asia. I'd been traveling for twenty hours straight and then I got to stand in line for a full hour waiting for an immigration agent to spend ten seconds looking at my passport photo to make sure it matched my face. Even the third world airports I've been through aren't that bad. There were even empty stations without agents. How much would it have cost to add a few more agents - $100? At the time t
Re: (Score:2)
Re: (Score:2)
They have most of the lines closed so they can back people up and watch who is starting to break out in a cold sweat.
Which is why most of the bad guys fly first/business class. Faster through the lines. Less chance of being observed.
Re: (Score:2)
Customs and immigration agents are Federal employees, and airport upgrades are state and private level. I don't know if an airport can simply pay Customs to supply more agents.
In any case, having another station open would cost a whole lot more than $100. There isn't a 24/7 pool of available customs agents who can simply show up for $50/hour, two hours minimum, when there happens to be a lot of people coming through. To man a station 24/7/365 (shouldn't that be 24/7/52.14?) takes a minimum of five peo
Re: (Score:2)
Apparently you've never had to run a payroll.
First, you're saying 8 hours a day. However, this only matters for international flights, and they can expect to get in pretty much whenever. We want to have coverage for more than when your flight happens to come in. Think 24 hours a day. That's something like 40 or 50 additional agents. Remember that number.
Second, people cost a lot more than you actually pay them. You have to pay FICA on top of that, there's likely to be benefits of some sort, and y
Here's the problem... (Score:2, Funny)
I think I found the problem, from the Department of State's own website:
"The Department of State is working with Oracle and Microsoft to implement system changes aimed at optimizing performance and addressing ongoing performance issues."
They're running Oracle on Windows.
Re:Here's the problem... (Score:5, Funny)
Or worse, they're running SQLServer on Sun boxes...
A Welcome to The Crash (Score:2)
I have arrived at the point where any crashes experienced by whatever State Department of whatever so called and self proclaimed Democratic Country (traitor mark here) are welcomed by me with the utmost glee. The more disruption, the more chances for a turnaround.
Large Databases? (Score:5, Insightful)
The article tries to wow us with the hugeness of the database, like this is a reason for the issues.
Yet the numbers quoted are not that big. Any modern PC isn't going to get too upset handling 75 million things. A real data center is going to sit there wondering what to do with the remaining 500TB of storage.
I don't doubt that there is some horrible flaw in the way the system was conceived that rendered it fragile, but whatever it is, it's nothing to do with the enormity of the problem, because it isn't very enormous.
Re: (Score:2)
75 million page-sized data objects (let's call it "text") would fit comfortably on my laptop hard drive*.
*2x2TB, I feel like I'm flapping my cock out when I'm sitting in a coffee bar playing world of tanks on it.
Re: (Score:2)
Ahh, Non free software. So I guess the FlexLM server went on the blink and is holding everything else up.
Re: (Score:2)
You owe me a new keyboard.
Re: (Score:2)
One thing I learned at an Oracle user group meeting: what you or I might think a large database is actually pretty darn small by industry standards. We discussed one where the activity pretty well maxed out write capacity on their disk drives.