Failed Win XP Upgrade Wipes Out UK Government Agency 731
Lurker McLurker writes "The BBC and the Register report that the UK Government's Department for Work and Pensions attempted to upgrade seven PCs from Windows 2000 to Windows XP, and ended up with BSODs on over 60,000 machines. I wonder if the National Health Service is regretting awarding Microsoft a £500 million contract now." The Guardian also has a good story.
The reason for the upgrade (Score:5, Funny)
We need to educate the decision makers (Score:5, Informative)
I have found that many MPs when questioned on anything related to technology simply say that "it is a complex issue", which to me isn't good enough when such huge amounts of money and significant impact on people's lives is involved.
There is a huge contract that'll be up for grabbs soon - EDS are preparing themselves to manage the UK national identity database and identity card scheme [itsecurity.com]. This is one we could lobby our representatives on to ensure they do it right..
Where to have the debate where it might be read by those who mater:
Free service to fax your MP [faxyourmp.com]
Boris [boris-johnson.com]
Richard Allan [richardallan.org.uk]
Tom Watson [tom-watson.co.uk]
Shaun Woodward [shaunwoodward.com]
Citing the recent and ongoing failures such as that cited in the article, and the UK Child support agency's computer failure [bbc.co.uk]. as well as the NHS computer system UK [bbc.co.uk]
Re:We need to educate the decision makers (Score:5, Funny)
No, no, no, this is the one we lobby them to employ EDS and Microsoft on!
If MPs are stupid enough to implement Blunkettcards we should at least get some entertainment out of it.
Best wishes,
Mike.
Re:We need to educate the decision makers (Score:3, Funny)
I do hope so! Blunkettcards can also be called 'Your inflexible friend' and should marketed with the slogan 'I know what's in your pocket.'
Best wishes,
Mike.
Re:We need to educate the decision makers (Score:5, Informative)
Free service to fax your MP
Can I take the opportunity to point out that faxyourmp is for UK citizens ONLY and should only be used to fax your own MP. It is not for international write-ins or mass lobbying.
Best wishes,
Mike.
Re:We need to educate the decision makers (Score:3, Insightful)
Why? If British people can be encouraged to interfere with the American political process [csmonitor.com], then why can't Americans do the same to the Brits?
Member of Parliament (Score:5, Funny)
Re:We need to educate the decision makers (Score:5, Informative)
A Member of Parliament, i.e. a member of the UK's primary legistlative body. Each represents an individual constituancy (area), and the government is formed by senior MPs of the party which has a majorty in Parliament (usually).
-Chris
Re:We need to educate the decision makers (Score:3, Funny)
Re:We need to educate the decision makers (Score:4, Interesting)
Is there another MP who's taken a clearer anti-ID card stance, and is prepared to discuss their positon so openly?
Re:We need to educate the decision makers (Score:3, Informative)
Re:All systems are prone to failure (Score:5, Insightful)
If someone can manage this by selecting the "wrong checkbox" then the system is broken by design.
Microsoft sell a complex system with the claim idiots can administer it. The DWP employ/contract idiots to administer a complex, but vital, system. Niether of these are "innocent parties".
Re:All systems are prone to failure (Score:3, Informative)
You can bollux up ANY operating system so it can't boot if you work hard enough.
Any decent Windows Admin should know (Score:5, Insightful)
This is first day stuff.
Re:The reason for the upgrade (Score:5, Funny)
Re:The reason for the upgrade (Score:3, Funny)
Dear Sirs:
I am a highly overpaid government employee with nothing better to do all day than read Slashdot and post AC comments, and frequent gambling and off-track betting sites. I and those of my many co-workers that were good enough to show up for work on a Friday and are not currently on a 3 hour lunch break and are looking at this on my monitor find the above post highly offensive and demand that you moderate it "flam
Re:The reason for the upgrade (Score:5, Insightful)
But a lot of them don't. I would say most state employees work their asses off doing pointless things, rather than screwing off. The problem is more with upper management than with the rank and file... though the problem does bleed over into the lower level employees because, after all, how long can you pour your energy into a task that you know only is neccesary because incompetent managers fail to streamline the operation and give you more real, productive work, before you start to take the job much less seriously?
So those petty state officials who shirk work do so as much due to being beat down, disillusioned, and tapped out as far as trying to do something about it in the face of a "front row" that doesn't like to listen to comments from their inferiors.
When I was working for the state, I considered myself very lucky to be involved in a project that was doing something meaningful, being productive and, while mistakes were made here and there, was relatively efficient overall. I could see how this was not the case in the departments working beside ours.
Eventually, though, the egos of the upper echelon managed to intrude even into our well defended (by caring managers) little island of fortitude and competance, and I had to say screw it. Now, unlike most of the rest of my friends that got laid off and sucked the government unemployement insurance tit, I am fending for myself with the money I saved by not buying useless crap.
So when people try to say I was overpayed at 60% of my fair private-industry salary, I don't shirk from the criticism. Yeah, the benefits were better than the private sector and the environment more permissive, but at least I didn't go looking for a handout like others so they could keep up the credit card payments for their DVD collections and car loan for their gas guzzling S.U.V.
At least I, one of those loathsome, lazy, state workers, had the good conscience not to apply my talents to better the carreer of a gaggle of idiots who aren't overseen adequately by the legislature that created their positions. If you want the state sector fixed, aim at the top. The clock punchers at the bottom are just a symptom of a management that preserves itself by not giving their underlings enough of a reason to revolt.
Re:The reason for the upgrade (Score:5, Insightful)
Looks like they got a deal; they got the version that also blocks viruses, worms, and abuse of Solitaire!
Writing article about Free iPod [tinyurl.com]. Please help out.
They probably wanted to block assholes who disguise 'Free iPod' links in the sigs. 'TinyUrl' my ass. If you want an iPod, ask your parents to raise your allowance. Otherwise, I heartily encourage you to fuck off.
Uh-oh... (Score:5, Interesting)
Re:Uh-oh... (Score:5, Insightful)
I'm sure the government has perfectly good reasons for continuing to hand contracts to EDS. It's just probably not a reason they want to tell you because it involves (bribery|nepotism|stupidity|all of the above)
Jedidiah.
Unit of time (Score:3, Funny)
= The ohno second - That minuscule fraction of time in which you realize that you've just made a BIG mistake.
This is typical of our government. (Score:5, Insightful)
I can imagine it now
Intern: "Sir, Microsoft have bought out Windows XP Service Pack 2. It's had numerous bug reports of dying pcs and software not working anymore. THIS is the time to upgrade to Windows XP, then upgrade to SP2 because windowsupdate won't stop bugging the hell out of us until we do!"
Boss: "You mean we could cock something up, and it might not even be our fault for a change?! Lets pay someone vast amounts of money to do it!"
The Gaurdian reports it was a week long outage. Now, I may be completely wrong here, but surely all they had to do was restore those pcs back to their previous Windows 2000 state using the daily backups they do... I mean, it's only common sense to do backups on such a critical syst...oh, wait, nevermind.
</cynical>
TCO costs rise scarily with Windows XP failures? (Score:5, Interesting)
Assume £8/hr employee. 40 hours of work a week. 60,000 unusable systems.
=> TCO increased by £19.2m for the 8 PCs they upgraded (before costs incurred fixing the problem)! £2m TCO per system for Windows XP eh? A clear example that Windows TCO can increase rather horribly if something goes wrong, and this was a standard upgrade. It's £320 per PC if you count all 60,000 systems - that's still horrendous.
Re:TCO costs rise scarily with Windows XP failures (Score:4, Insightful)
I once knew a bean-counter (quite senior) on nearly 3 times my engineer's salary. He was sat there in front of a spreadsheet adding up a column of numbers on a pocket calculator.
Welcome to the UK Public Sector. That was your tax money.
Re:TCO costs rise scarily with Windows XP failures (Score:5, Insightful)
In my experience (having worked for both) in terms of inefficieny and stupidity, there's only one thing worse than the British Public sector and that's the British Private sector.
My company used to be part of a large public sector concern and was sold off. Since then we seem to spend nearly of our time/money:
Changing company logo and name every 6-12 months
Adding a new problem management system which we have to learn every 6 months (we currently have about 5 each of which was supposed to replace all the others).
Paying huge bonuses to upper managent.
Paying huge car allowances to middle management including those who refuse to drive.
Not giving any rises under the so-called performance related pay scheme for 4 years despite meeting profit targets because all the money has gone on the above 2 items.
Making skilled people redundant then recruiting at vast expense people with the same skills 2 months later.
Making skilled people redundant then reemploying them at twice the pay as contractors for the next 2 years because they're still needed.
Repeatedly shuffling kit from datacenter to datacenter around the country at vast expense and disruption to our customers.
Ordering expensive buffets for management meetings , 95%+ of which get thrown away.
Managers having a schedule involving meetings all over the country which means that they spend about 25 hours out of 40 driving.
Managers refusing to use video-conferencing for meetings even in the light of the above.
How many of these things happened when I was in the public sector? Virtually none. We didn't have the money to throw around on such things. We were forced to be efficient.
Also if this private sector company I'm referring to was atypically inefficient, presumably it would do so badly it would collapse or be taken over. So this implies that many private sector companies are like this.
It's very easy to slag off the public sector if you use stereotypes, generalizations and distortions.
Re:TCO costs rise scarily with Windows XP failures (Score:3, Informative)
Re:TCO costs rise scarily with Windows XP failures (Score:5, Insightful)
Yes. It's not like the upgrade could detect the version of the program it's being applied to, and only install if the version matches the version it is intended for. That is completely unheard of, and would be impossible technically.
This was sarcasm, FYI.
This situation is more analogous to a wrong signal causing the door to open and then jam. And yes, such a door manufacturer deserves to be blamed.
Re:TCO costs rise scarily with Windows XP failures (Score:3, Informative)
This sounds like they were pushing out the upgrade via SMS. Checking that the upgrade was on an appropriate system here would not have mattered since the upgrade path from win2k to WinXP is legitimate. This sounds more like sysadmins instead of applying to a custom collection applying to the "All Systems" container. The real question her
Re:TCO costs rise scarily with Windows XP failures (Score:5, Insightful)
Given, they should actually have an install script that checks the OS before it actually dumps the install package on there, but hey.
Not normally an MS apologist, but this isn't really Microsoft's problem. It's the contracted company that made the update package failing to ascribe it to the right download group.
So, the analogy. It's like some perfectly good system being installed, and someone presses the button marked 'open all doors' instead of simply open door 7.
I don't see anyone really blaming the door manufacturer here (Microsoft or the contractors), although I'd hazard a guess that the person who skipped over the part of the process that said 'double check the groups you assign this patch to' will be sorely chastised...
Re:TCO costs rise scarily with Windows XP failures (Score:3, Insightful)
Company polict stated that everyone should always turn off there PC's when they left for the day and you'd get moaned at if you didn't. The Radia team told everyone they must keep their PC's on at all times but this was never company policy.
Every morning it would take 20mins or so for Radia to install all the ni
It's about processes (Score:3, Insightful)
It's this willingness to say "Localised error. That's all. Nothing to see here" that gives IT it's bad reputation. With properly designed processes and appropriate tools, localised error cannot have catastrophic consequences. In a system like this, I can see no excuse for pushing so
Re:It's about processes (Score:3, Insightful)
Excactly, and IT earned every bit of it. No one wants to pay for processes, no one wants to expend the extra effort for processes, and no one does. People in IT are more comfortable taking the intellectually lazy route and, because it works 80% of the time, they become quite comfortable doing it. For that other 20% or whatever, they figure out how to rationalize it as a "software glitch", e
Re:TCO costs rise scarily with Windows XP failures (Score:5, Interesting)
Re:TCO costs rise scarily with Windows XP failures (Score:3, Insightful)
However, I don't know any reports which consider Total Cost of Ownership Assuming Your IT Department Is A Bunch of Blathering Idiots. Most seem to assume a certain degree of competence.
FAT CLIENT (Score:3, Interesting)
This is what happens when you have a fat client. There's a lot in a fat client. A lot to go wrong, a lot to be insecure. It therefore needs a lot of looking after. Many updates, many risks. Multiply by many desktops and it only becomes manageable by central updates. Ce
Re:FAT CLIENT (Score:3, Insightful)
Cheers,
Adolfo
Re:FAT CLIENT (Score:3, Informative)
Re:This is typical of our government. (Score:5, Interesting)
Only theregister appears to talk about Win2k and XP, so lets see what they're saying.
So if this is true then EDS pushed out a partial upgrade. Now come on, if you installed 75% of a new distro over an old one then rebooted would you blame Redhat because it didn't work?
Or there's the other version
So again EDS pushed out XP patches, overwriting Win2k files and the machines crashed
Not really surprising if you overwrite parts of an OS with files from a different OS that there is a mass crash, but folks, this is an EDS fuckup not really a problem with Windows.
Of course theregister could be wrong. It might happen. Heh.
Re:This is typical of our government. (Score:3, Insightful)
Guess what? EDS chose to do it themselves using a third party product rather than use the much more mature and safe existing update tools.
Now who's fault is that?
Re:With Linux, it would be harder to do ... (Score:3, Interesting)
Second, The machines wouldn't boot, therefore there is no way to run any kind of script to fix the problem, thus your third solution is likely what happened in this case as well. However, it takes some time to manually go to 60,000 machines and fix them, even if it only takes 5 minutes per machine.
The exact same thing would have happened
Re:This is typical of our government. (Score:3, Insightful)
or have I missed something?
oh hey what's going on with this upgrade (Score:4, Funny)
Too slow. (Score:4, Informative)
It's certainly bad PR for Microsoft though, perhaps this will serve as a wake-up call to other governments that "other options" are out there.
Re:Too slow. (Score:5, Insightful)
Re:Too slow. (Score:3, Insightful)
It's still Microsoft's fault, because they designed a system that accepts updates for the wrong system, and after that update is installed, it's damned near impossible to back it out. EDS has fault here too, but let's face
Re:Too slow. (Score:3, Insightful)
Somebody else in the thread mentioned this - if you overwrite your Linux kernel with a botched version, your system's hosed. If you didn't keep a backup, it's damned near impossible to back it out.
Nobody can protect an incompetent admin from him / herself.
Re:Too slow. (Score:5, Insightful)
Now, assume Microsoft bails EDS out, and there is no reason why not, because you can bet they'll send a bunch of temps to every DWP office at EDS' expense if they have too. In a nutshell, Microsoft gets a PR coup: "We've just bailed out out a leading *cough* solution provider! Now imagine that had been, say, a Linux deployment... Who could EDS have called then?" Given the excellent grasp of PR, spin and FUD Microsoft has, I don't think this is going to help break the Microsoft stranglehold at all.
Re:Too slow. (Score:5, Insightful)
They could have called Novell or IBM.
Apart from that though - any setup can be screwed-up by an admin, no currently available OS can protect you from that. So for a TCO estimate at least we would have to look at the total loss due to screw-ups like this, and weigh them with the number of installations. Using a single data point can't be valid. That said, my gut feeling is that Linux provides considerably better TCO.
Re:Too slow. (Score:3, Funny)
This is actually a really good question. One thing I've found in Linux support is much of the software (the new software raid as an example) isn't clearly documented and when you do run into serious problems beyond a few simple things to try people generally seem clueless - even very experienced people. I blame a lot of this on constantly moving support targets (the day you document one issue, and its solution there have
I wouldn't want to be in their shoes now (Score:3, Funny)
EDS again (Score:5, Informative)
Re:EDS again (Score:5, Interesting)
Yet the government keep awarding them [EDS] contracts. Why?
I don't know, but I do recall an article about IBM refusing to tender for UK.gov contracts: apparently it was too costly, and too risky - you could spend millions only to not get the tender, and IBM felt that the chance of getting the tender awarded to IBM was too small. So... I'd suggest either it's too costly to play so players are dropping out (the reasonably answer), or someone in government really loves EDS, and IBM know it (the tinfoil hat answer).
Living in the UK, I'm minded to go for option 2.
Re:EDS again (Score:5, Insightful)
Because Accenture is the other choice!
This sort of cockup would have been impossable with the ex Arther Anderson crowd. They would still be struggling to get the shrink wrap off the CDs without wrinkling thier suits.
Seriously the problem is government procurement procedures. The contract goes to the lowest bidder and a record of past f****ups is not taken into account.
Re:EDS again (Score:5, Interesting)
There is a huge hue and cry (outrageous exclamation of disgust and anger) over mismanagment and eggregious spending in government contracts. Having worked in the sector, I'm somewhat familiar. The contractor I worked for made sure there was no waste, fraud, or abuse. However, it spent 10 times as much as the job required, just to do this. The obvious choice for our firm was it would have been far cheaper to run things by GAAP (Generally Accepted Accounting Practices)(the private sector accounting standards), and have both a nice large internal audit division and "internal affairs" watchdog enforcement. Alas, most governments are not run this way, and if they are, they devolve into the current format due to political expediency.
I have friends that work for EDS and they comment on the kinds of hoops they have to jump through just to do simple stuff. They've built up a rather large experience pool in doing this hoop-jumping, so they can do contracts cheaper than some other companies.
EDS also tends to run things according to CMM levels whenever they're developing things, so at least if there's a mess-up (as there obviously was here), there will be some kind of follow-through to improve the process of doing this kind of work. EDS's management doesn't want the black eye any more than the government or Microsoft do, but they'll spend the money to make sure it doesn't happen the same way again. There is, after all, no way to prevent all errors, but I give them credit for trying most of the time.
Re:EDS again (Score:3, Insightful)
The idea that you pay one company to come up with a box of requirements then send it out to tender, and get several boxes back from a few large companies like EDS. Then these get send off to the company contracted to deal with the subcontracting/tendering process. A haggling process commences between bunches of lawyers on both sides resulting in usually only one or two possibilities the cheapest one is then selected and fucks it up. Now
Re:EDS again (Score:3, Insightful)
Please realize that I'm not defending them. I'm just pointing out that, as someone who works in IT, management never sees it when things go flawlessly, but they will not hesitate to throw your ass to the wolves should something go wrong.
*sigh* (Score:3, Insightful)
Come on now (Score:5, Insightful)
Hey! let's be fair here, ok? (Score:5, Insightful)
So, admin stupidity can also be blamed on MS, it's part of the TCO studies that make the decision to buy MS.
Aside from that, a point-and-click update cannot fail so miserably. A script made by the admin, of course should, because you can assume that someone smart (and bold) enoguh to make a little script should be responsible for their decisions. Some guy clicking checkboxes shouldn't be allowed by those means to break 60000 computers, through a
GUIs for dummies should have enough checks to prevent such underiable effects, they have a sufficiently constrained domain to be able to do so. If the guy wanted to do a legal task that the tools dosnt' allow, he could always write some Visual Basic Script, and then he would be on his own. Bringing down an organization by mis-clicking checkboxes is responsability of the guy that provided the checkboxes, too.
What should be done first... (Score:2, Flamebait)
Next, they should setup a similar Linux environment and see how a similar upgrade would have gone. This should be done before the Linux zealots declare that theirs would have been any better.
In the meantime, let M$ and its cronnies be prepared for the wrath of Linux promoters.
Cb..
Re:What should be done first... (Score:3, Insightful)
EDS now stands for... (Score:5, Funny)
Nooo! (Score:5, Funny)
What the heck did they do? (Score:4, Funny)
I RTFA and all I see is a money discussion, not a technical discussion. I would speculate that an SMS or Zenworks push or somthing similar which was supposed to be restriced to the 7 PC's went almost everywhere. It might be a fair bet that the remaining 20,000 might have been upgraded too if those people had been at work and turned on their computers. IT Computer management tools give the department much power, which could do plenty of damage in the wrong hands.
For those not reading the article... (Score:5, Informative)
Jon.
EDS managed upgrade--Altiris? (Score:5, Interesting)
It's a great set of tools--we own it at work and managed our own Win2k -> WinXP upgrade using the PC Transplant and Deployment Server tools, but can massively bone you if you don't do enough testing. PC Transplant, in particular, can hurt if you--that's the application that lifts your profile off of one PC and slaps it down on another, so that you don't have to re-configure your Exchange settings, Office personalizations, backup documents and application settings and bookmarks, and a whole mess of other things. When doing an OS migration, if you don't design your personality transplant template correctly, you can end up with all kinds of Win2k-specific settings stuffed into your WinXP profile, which can lead to all kinds of crazy-ass problems.
RTFA! (Score:5, Informative)
Brilliant work on the part of EDS, trying to patch the wrong systems, lord only knows what can happen then.
You could force an XPSP2 onto a 2k machine... would you still blame Microsoft for it? That seems to be the case here, EDS screwed up, and of course it's Microsoft's fault in the eyes of
Re:RTFA! (Score:4, Insightful)
Contractor (Score:3, Insightful)
The installation and update of operating systems is so easy any more, a blind one armed monkey masturbating could do it.
I've worked with EDS people, and the one armed monkey would be a godsend compared to most of them that I've had the "fortune" of working with...
Avoidable blunder (Score:5, Funny)
Wrong! (Score:5, Informative)
the UK Government's Department for Work and Pensions attempted to upgrade seven PCs from Windows 2000 to Windows XP, and ended up with BSODs on over 60,000 machines.
In actual fact, the Register quotes:
According to one, a limited network upgrade from Windows 2000 to Windows XP was taking place, but instead of this taking place on only a small number of the target machines, all the clients connected to the network received a partial, but fatal, 'upgrade.'
and then below it:
Another source says that the DWP was trialing Windows XP on a small number ("about seven") of machines. "EDS were going to apply a patch to these, unfortunately the request was made to apply it live and it was rolled out across the estate, which hit around 80 per cent of the Win2k desktops.
So, by merging them you get the following story:
There was a trial of seven PC's, instead of patching only those seven, the request to roll it out was accidently performed and every computer attempted to install a botched version of XP.
Somewhat slightly different to the Slashdot version wouldn't you say?
In addition, I'm pretty sure that if you accidently deployed a botched version of the linux kernel then it too would probably have a similar effect.
what the hell went on??? (Score:3, Interesting)
You guys are amazing! (Score:3, Insightful)
I wish I could take one of you Linux "experts" up on your idea. "Here, upgrade these 2000 PCs, all of which are from different manufacturers and different configurations, to Linux. I need it done in the off hours and I need everything to work like it did before.".
*crickets*
Of course someone will reply and say "ok!" knowing it won't happen. It's not because I don't have the ability to make that decision but it's because I know better than to get real information/insight about IT from most /. posters.
It's painfully obvious that a scant few here actually have a clue about running a business that relies on IT. It's more than ripping CDs and DVDs kids. Sure, the company that did the mistake is at fault but the problem is not in the chosen OS, it's in the chosen technicians and management.
Re:You guys are amazing! (Score:5, Insightful)
You know that (re)installing Windows on a large number of systems of different types, for example when an upgrade fails, is a total fucking nightmare, yes?
At least Linux comes with 99% of drivers pre-installed. With Windows you have to find them on the net first, then find some way of getting them to the target system (because you don't have a NIC driver, remember?).
Re:You guys are amazing! (Score:5, Informative)
As for managing large networks of desktops, that's another very different matter. Not many people have high-level experience doing that.
My network, for example, is only thirty machines. Hardly huge. In fact, it gives me the opposite perspective on a lot of issues, because I find many of the large-site friendly features of Windows networks utterly useless for a small site, and no small-site friendly managability features to compensate.
Personally, I've trialled XP at work as a possible upgrade for our 9x machines, and come to the conclusion that it's not worth the pain. It might be good if you have the management tools, a dedicated test network, and an admin team dedicated to designing and rolling out updates. For small sites, however, it's pure hell. Even controlling how the clients update themselves is hard without an extra server to do the job. I also found accessible information for small-site management to be very thin on the ground.
We're now using thin clients for some of our network, and seeing very good results. Yes, they're Linux based - MS looked good until we figured in the CALs and the isssues with NT-based terminal server security. I'm far from floored by the results with Linux - the bugs, oh, the bugs, I'm drowning in stupid f***ing bugs. There's also more than a little totally retarded design, and the classic issues with no two apps having the same open/save dialog.
That said, for our basic users the results have been very good. They need little support, hardware and software costs are both low, and things generally run very smoothly. Trials with more demanding users aren't going as well (see above rant about bugs and bad design), but current development in the OS is addressing most of the issues I've run into and I expect to be able to move the 9x users across to the thin clients mid-late next year.
I do agree with you that managing a large collection of Linux desktops would probably be pure hell. It's awful to even think about, frankly, especially upgrades. *shudder*. My solution would be to simply not use desktops, but instead move most users to department level thin client services hanging off a redundant set of beefy servers. I'd use LDAP to store user and sytem information (yes, much like AD) as I currently do on my network. For many users, such a setup can be expected to work very well, and dramatically reduces the admin nightmare compared to Linux desktops. I also wouldn't even try to migrate all users to Linux - only basic users for whom it would work well, such as those who only need email, a browser, a word processor, and access to a couple of specific in-house apps.
As for migration - I can't possibly imagine how it could be done in a sane way. I suspect a lot of custom tools would have to be written, the migration would need to be a rolling one, and there would need to be a lot of staff on hand to handle glitches. That doesn't sound like fun to me.
The worst part of moving my users over to the thin clients was migrating their data and settings. That despite the fact that almost all of it was already on the servers, and their systems were pretty basic and very uniform. Doing it in a large company wouldn't be nice.
Hope they all loose their jobs tomorrow (Score:3, Funny)
Re:Hope they all loose their jobs tomorrow (Score:5, Insightful)
They emigrated, most likely. One of the problems with incompetence is that it's self-reinforcing, the competent get more and more fed up with having to deal with incompetence all day and find something better to do with their time.
Re:Hope they all loose their jobs tomorrow (Score:5, Insightful)
The funniest thing.. (Score:5, Funny)
What a big surprise (Score:3, Insightful)
"This patch caused the desktops to BSOD and made recovery rather tricky as they couldn't boot to pick any further patches or recalls. I gather that MS consultants have been flown in from the US to clear up the mess."
So, even more of the money I pay in tax is being diverted to M$ then...
Ah. I've got the picture now... (Score:3)
As much as I would like it to be, it doesn't seem like a "Microsoft" problem exactly and were a parallel Linux situation have happened I'm not sure anything less would have happened... well I guess it would have to depend on a number of things -- for example, if it were an RPM-using distro on the desktops and the wrong RPMs were sent to ALL machines instead of the select few, the machines for which the upgrades were unsuitable would have simply failed due to dependencies unless the --force option were used... okay I'm rambling now but basically, I don't see it as a Microsoft problem as much as I see it as a misuse of tools.
The TCO of their MS installation just went up though... and they shouldn't exclude the cost of firing, hiring and retraining either.
Come on! (Score:4, Informative)
1. The patch they tried to update with wasn't a complete one for an OS upgrade.
2. Then they deployed it to their entire network by mistake.
This interesting piece of information can be gathered by RTFA.
I wonder what would happen to, say, Linux boxes if they had 60,000 and they applied an incomplete kernel patch?
Maybe some... thing... would panic?
Perhaps I'm just missing something here.... (Score:5, Insightful)
Windows? Or EDS? (Score:3, Insightful)
As much as I'm sure the zealots among us would like to make this seem like a Windows failure, it looks like it's more of an example of how outsourcing leads to disconnected, incompetent, and unmotivated IT staff. And that, of course, leads to mishaps like this.
Either way, if you work for a company that brings EDS in house in any way, drop your shit and run. And don't look back. The flash could be blinding.
How Stuff Works (Score:3, Funny)
Jones agreed, "I still have plenty of blank cheques. My pen is at room temperature."
Fundamental Architectural Issue Here (Score:3, Insightful)
The fundamental error here is deep seated and architectural - they have 80,000 user interface devices which are stateful. By putting the wrong device on the desktop they have set this situation up.
In the olden days when clerks in government agencies used green screens this problem wouldn't happen. If a green screen failed, it would be replaced as a FRU. Today's equivalent is something like a SunRay - the user interface device holds only enough configuration to bootstrap itself and, again, is a FRU.
The situation at the DWP is different: the user interface device is a stateful device which holds configuration itself, and requires this configuration to be consistent before it gets enough connecticity to be remotely managed. The toolkits discussed, which are used to push config around these UI devices, are probably most excellent, but there should be no need for this sort of mularky.
So while I don't necessarily blame Microsoft for this incident, I do blame them for creating a monoculture where this sort of architecture is deployed. I expect the trials underway in government using SunRay devices as the user interface will be watched with more interest after this debacle.
A final question - how on earth do DWP recover 60,000 unbootable PCs?
They shouldn't have upgraded (Score:3, Insightful)
EDS is a joke at taxpayers expense (Score:3)
It's well known that the UK government are in the pocket of EDS and Microsoft. The worst thing is that it's not intentional. The people in charge of making these decisions are complete non-techies and haven't heard of any IT company that aren't a regular in the new headlines of the FT. It's not corruption, it's basically a lack of education.
Phillip.
Not a nail for Microsoft. (Score:5, Interesting)
If you read the register article, it says that they were attempting to only push the update out to 7 PCs, but it actually went to all 60,000.
I would imagine they were using something like Microsofts SMS services or Bigfix to push out packages, and simply selected push out to all instead of a test community.
I don't think this is a nail in Microsofts coffin, I have seen similar things happen in the mainframe world where patches intended for dev hit live production systems with similar bad consequences. It has to count as a bad day at the office for the person pushing the button though.
It also highlights the difficulty in pushing out big updates to major networks of PCs, be they running Windows or Linux. The complexity of moving from Win NT to XP has proved so complex in my organisation that for the future Longhorn upgarde and beyond we are now looking to Citrix to allow the migrations of applications across servers and essentially use the PC as a thin client for all but core office and email apps.
Re:Not a nail for Microsoft. (Score:3, Insightful)
Re:Not a nail for Microsoft. (Score:5, Insightful)
Doesn't anyone do risk analysis anymore?
Re:Not a nail for Microsoft. (Score:5, Insightful)
It's true that Microsoft's robustness is rather mirage-like, but there's a thing called human error, and that can bring down any system. All the software did was follow human instructions, after all: that's why we need IT people with brains to decide who is doing what.
However, PXE boot and a server with HDD images ready would've been helpful...
oh yea (Score:3, Funny)
When a government ends up with BSODs on 60000 computers, it can't be good for Microsoft.
Yea, I can just see them going bankrupt over this. Their coffin was half closed before, but now they're bound to be pennystock.
Re:Another nail? (Score:5, Funny)
No, but that doesn't necessarily mean it's bad for the rest of us!
Let's hope Congress plans to upgrade soon!
See? Even Microsoft is good for something!
Re:Another nail? (Score:5, Funny)
"Do you want to update the machines on your network now?"
[Accept]
No cancel button.
--
Wiki de Ciencia Ficcion y Fantasia, un cuento por Fly. [uchile.cl]
Re:Another nail? (Score:4, Interesting)
You'd probably be retired now! Pity you chose long hair, and have another 40 years of work to go.
Re:Another nail? (Score:3, Interesting)
You'd probably be retired now! Pity you chose long hair, and have another 40 years of work to go.
I get to do cool stuff with UNIX nowadays. 40 years of cool stuff is better than becoming an EDS pointy-hair for 4 years and having to learn IBM JCL.
Re:If this was in the private sector... (Score:3, Insightful)
Re:umm.. (Score:5, Informative)
The question about all of this that I am left with is, how did the patch even install? Microsoft has had sanity checking on their patches for ages, checking not only the Windows version, but even service pack levels and any other prerequisites. Ever tried installing a patch intended for IE6-SP1 over plain IE6 for example? I'm assuming that this is some custom patch rolled by EDS, rather than an official Microsoft one downloadable by all and sundry. Still, the story appears to have made it onto UK prime time news, so no doubt more details will emerge...
Re:Not Microsofts fault, in this case (Score:5, Interesting)
My co-worker and I spent the next period AMAZED that Windows networking even worked at all. The system of domain controllers and WINS servers and browse lists and host files... it's too byzantine to be believed. There is, without doubt, a corporate network somewhere that could be comopletely undone by someone opening a wireless laptop in the wrong place at the wrong time. Add Windows XP and the attendant SP2 fun they're having and you get chaos.
Yes, those delightful folks at EDS are the chimps in this scenario, but Microsoft's products are definitely the defective Uzi. And I note that the BBC News article studiously avoided mentioning either of them. Hmm... Microsoft wouldn't be doing everything it can to tamp down this PR disaster, would it?
Naaah!