Emory University SCCM Server Accidentally Reformats All Computers Campus-wide

acidradio writes: "Somehow the SCCM application and image deployment server at Emory University in Atlanta accidentally started to repartition, reformat then install a new image of Windows 7 onto all university-managed computers. By the time this was discovered the SCCM server had managed to repartition and reformat itself. This was likely an accident. But what if it weren't? Could this have shed light on a possibly huge vulnerability in large enterprise organizations that rely heavily on automated software deployment packages like SCCM?"

Re:Did the backup and restore work?

[wisnoskij]

This is University. Based on my experience with University IT, there will be loads and loads of important data that you are legally obligated to NOT do that to. It cannot leave one specific room, in any form.

Normally, the computers are still contacted to the network and the Internet, but everyone using them must know NOT to copy any of these files off of C.

Re:An...accident..?

[bruce_the_loon]

This isn't the update server section of System Center (WSUS), it's the machine deployment system (Configuration Manager), and it can quite easily do this if left as-is out of the box with multiple technicians on it. And it can be done accidentally.

Here's the scenario as it likely happened.

• Technician finished a master PC install task sequence and tested with one PC. Now he is ready to deploy to his computer lab.
• There are two options for failure here. SCCM allows for collections of machines to be built for all purposes (data gathering, deployments etc), so he probably puts a quick group together and gets that step wrong and the collection includes all computers in the AD tree. One of our technicians did this after two years of using SCCM regularly.
• Or he goes hunting for an existing collection and ends up selecting the default All Systems collection which includes everything. If there are a lot of collections or his is named too similarly.
• After another hundred odd clicks, he hits deploy and SCCM sends a message to the client service on all computers in the selected collection to run the new deployment task sequence. Including the SCCM server because it also has a client and is in All Systems collection or gathered in an incorrectly specified collection.
• Each PC then downloads the image, reboots and wipes itself with the image. The server, also in the collection, will do the same at some point.

We've had two near-misses with misconfigured collections and one hit with a different problem* which cannot have happened in this case. SCCM isn't the most intuitive user interface and if you're being pressured by users or trying to get out of the door for the weekend, you can stuff it up easily.

Our solution was to restrict access to the built-in collections and to build collections per computer lab which are presented as read-only to the technicians. And then gave them a day of lectures. It sort of works.

* The other problem was caused by image dumping with Ghost of an image that was sysprepped, but had the SCCM client still installed on the image. Because of that, several dozen PCs had clients with the same client ID, like the Windows GUID, but separate and not cleared by a sysprep. The technician later built a SCCM image and deployed it correctly to one PC in a personal collection. Unfortunately SCCM populated the deployment list based on the client ID of the PC in the list and hit quite a few overnight. Luckily a lot of the machines in the batch were off overnight. I don't think this is the case because it hit the server too and that would have received a new client install during the SCCM installation.

Re:Surprisingly Infrequent

[Anonymous Coward]

Pah, this is is small fry. If you want to see a proper clusterfuck check out what happened at the Dept for Work and Pensions in the UK in 2004/2005 (I forget re exact year). Somebody at EDS sent out a test update to 87,000 computers rather than the 200 or so it was supposed to gone to. They got the negation wrong in the logic and so the real network was targeted.

Just to really, really screw things up, the update partially failed so the machines were left in an inconsistent state and couldn't be used in any shape. As I heard it, the people who wrote the distribution system from Microsoft were on the first flight over once shit had hit fan. 10,000s of users administering benefits and other vital things to the uk public simply sat at their desks twiddling their thumbs. To be fair to Microsoft (not a phrase I would normally use) they pulled eds's balls out if the fire. They managed to get a small custom made bootstrapping image out that apparently loaded more and more stuff in to get the machines back to working order. I'm told by people in the know it was a first rate job that cemented Microsofts position in UK govt.

The head of the DWP also did a sterling job of keeping the lid on and stopped MP's quite appreciating how bad it was.

The only thing that really saved them was the fact that all the hardwork was really done on elderly ICL mainframes so the lack of Windows clients was not that big a problem. I suspect that they sre still there.

Nice to know the UK still leads on something.

Rob