Passport Database Outage Leaves Thousands Stranded

linuxwrangler (582055) writes Job interviews missed, work and wedding plans disrupted, children unable to fly home with their adoptive parents. All this disruption is due to a outage involving the passport and visa processing database at the U.S. State Department. The problems have been ongoing since July 19 and the best estimate for repair is "soon." The system "crashed shortly after maintenance."

Replication != Backup

[Anonymous Coward]

From their Q&A:

Q: Why wasn’t there a back-up server?
Back-up capability and redundancy are built into the system. The upgrade affected our current processing capability, in part because it interfered with the smooth interoperability of redundant nodes.

We don't need backups, the data is replicated, we're cool.

Re:Change management fail

[dave562]

As much as I am not a fan of government regulation, my professional experience has shown me that the only time people get IT anywhere close to right is when there is a risk of financial penalty involved in getting it wrong. Regulation seems to be the only solution to people working for peanuts. The people who work for peanuts make mistakes. If those mistakes cost the company more than the company saves by hiring those people, they will not hire those people.

Out of all of the industries that I have worked with, the financial services industries seem to be the most together. They are not perfect, but the penalties associated with losing customer data makes them more careful.

Re:Change management fail

[l0n3s0m3phr34k]

This isn't always the case. A company can save money via outsourcing IT infrustructure if they go with the right vendor. VMWare, virtual servers, proper fail-overs, big multi-core blade racks were the VM is still more powerful than your original server and still costs less...but of course I work at HP in the Enterprise Services so the level I'm talking about probably isn't affordable for a "small company". We have VERY specific steps for everything, our "runbooks" detail everything from server configs, hardware, rack enclosures, port layouts, and responsible parties to contact for each part if it fails. When you have a rack of blades, it's far easier to snapshot, launch then test,,,we always have a "backup" in a hot image ready to go if anything fails. Often I'm working with 5-15 people spread across the globe all doing different functions (unix admin, wintel, recovery, netops, etc) but we rarely have any "HP owned" customer-impacting outages. Of course my major clients are airlines so it's all tightly regulated; your individual milage may vary LOL.

Re:Change management fail

[ShanghaiBill]

simple yes or no questions

It is only simple because you speak English. You need to widen your cultural perspectives. In other languages, and other cultures, it is not so simple. For instance, Chinese does not even have the words "yes" and "no". If you ask a Chinese speaker if they have a pen, they will answer "have" or "not have". If you ask them if they are going to lunch, they will answer "going" or "not going". There is no such thing as a "yes or no question" in Chinese, and culturally, Chinese are much more direct than Indians or Japanese.