Emory University SCCM Server Accidentally Reformats All Computers Campus-wide 564
acidradio writes: "Somehow the SCCM application and image deployment server at Emory University in Atlanta accidentally started to repartition, reformat then install a new image of Windows 7 onto all university-managed computers. By the time this was discovered the SCCM server had managed to repartition and reformat itself. This was likely an accident. But what if it weren't? Could this have shed light on a possibly huge vulnerability in large enterprise organizations that rely heavily on automated software deployment packages like SCCM?"
Cool (Score:5, Funny)
Sounds like a good way to get rid of Malware
Re: (Score:2)
inb4 the whole system was compromised
Re:Cool (Score:5, Interesting)
Unfortunately, SCCM [wikipedia.org] also supports Linux and Mac OSX clients. I wonder whether it tried to install Windows 7 on them also? Users would be really pissed to discover their Mac/Linux box was now lurching under Windows...
Re: (Score:3)
Only if their machine was part of the University's Active Directory infrastructure, as far as I know. Just being on the same network wouldn't be enough.
Re:Cool (Score:5, Insightful)
IPMI and other "lights-out" (Score:3)
Don't underestimate IPMI and its equivalent (what was Intel's name for their proprietary alternative? ME ?)
With this kind of technologies, you have a small mini-embed system in the motherboard, which talk over a TCP/IP network and provides all functionnality (including wake-on-lan, including shutdown, including reboots (AND specifying what resource to reboot to - like starting PXE or emulated-over-network USB driver), including VNC remoting, and Serial-over-IP (for some server remoting).
Think of all the nic
Re: Cool (Score:5, Funny)
I worked at Emory for years and I have no doubts this was sheer incompetence not sabotage.
Re: Cool (Score:5, Funny)
mac systems may not even boot with Partition types (Score:3)
mac systems may not even boot with the old Partition tables that are needed for older NON EFI systems that windows runs on.
also the Mac os Recovery Partition may even be wiped out.
Re: (Score:2)
apple uses EFI / UEFI.
older Pc's don't have it and in some cases with pc that have it have it turned off so they can boot XP, Windows 7 (in some cases), disk encryption, and other stuff.
32bit os's yes some places put 32 bit loads on systems with 4gb or more ram in them.
Re: (Score:3)
apple uses EFI / UEFI.
older Pc's don't have it and in some cases with pc that have it have it turned off so they can boot XP, Windows 7 (in some cases), disk encryption, and other stuff.
32bit os's yes some places put 32 bit loads on systems with 4gb or more ram in them.
Well in some cases that's for the best. 4GiB or even 8GiB is the standard amount of memory of many computers, not that one would want any thing less anyway (it wouldn't lower the costs noticeable).
And there are reasons to run 32 bit OS installations on 64 bit systems, I've have to use a separate 32 bit Windows installation in a VM to run some important programs.
Re: (Score:2)
AFIK the heterogeneous client on Linux/Mac only handle installation/removal/inventory/monitoring of applications and not pre-loading of a complete OS.
Of course, this may have changed since I last worked with it but there are some inherent problems to get that to work under Linux/Mac et al.
Re: (Score:3)
OSD is only supported on Windows. (source: I'm a CM 2012 admin).
Re:Cool (Score:4, Funny)
What no tape backup?
Look at all the money we saved!
Re:Cool (Score:5, Interesting)
That's what some universities actually do. They have a custom built dual-boot OS partition image (Linux + Windows) will all the standard applications that have been licensed and required for lab use (Mathematica, Microsoft Word, Firefox, Opera). This image gets stomped onto the drive of every idle system every night. So even if some spyware installs itself overnight, it gets overwritten.
backups (Score:3)
Time to test those backups!
Re:backups (Score:4, Funny)
That's what RAID-5 is for, jeez.
Re: (Score:3)
That's what RAID-5 is for, jeez.
Um, you do know that RAID is designed to protect against hardware errors and can do nothing if a command is given at the OS level to write over stuff, don't you?
SCCM server reformats itself? (Score:5, Funny)
Kind of sounds like a snake eating its tail....
Re: (Score:3)
Re: SCCM server reformats itself? (Score:2)
Honestly it sounds more like it wasn't a mistake, and something malicious if the server reformatted itself. To, you know, cover your tracks.
Re: (Score:3)
The server didn't merely reformat itself. He installed *windows*.
It's obviously a murder-suicide case.
Configuration deplorement (Score:5, Funny)
Centralized Control... (Score:3)
The problem with centralized control is that the center can give any commands it wants...
Re: (Score:2)
The problem with centralized control is that the center can give any commands it wants...
Somebody probably should have told that to the Bolsheviks...
Sounds like IT incompetence (Score:5, Insightful)
Re:Sounds like IT incompetence (Score:5, Insightful)
Assuming it was just a mistake and not malicious ...
Probably not. This shit happens, and that person who did it will never do something like this again. Have you ever made a massive, expensive mistake?
I have, I was 19 years old and cost my company nearly a million dollars due to a silly misconfiguration. After I discovered it, corrected the error and notified my boss, I spent most of the night throwing up. The next morning, after everyone in the company (only 15 people or so) knew what happened, and I walked through the halls on the way to the meeting with the owner and my boss, I thought I'd pass out. As I walked into the Owner's office I didn't even bother to sit down, expecting a fairly short conversation. I was asked to sit down while my boss had this very stern look on his face. So I did, cost them that much money, I can do what they ask.
The owner than proceeded to tell me the story of how, when working for a certain Germany car company doing CICS programming, he made a mistake that screwed up a production line and cost the company several million dollars. He knew exactly how I felt, and he knew that it would never happen again because I had already punished myself more than he possibly could.
If they fire the person who did this, they just wasted the whole event. The person learned their lesson and will be extremely cautious in the future. Firing them now just means someone else will get to reap the benefits of this experience, and thats pretty stupid.
People make mistakes, and in this case the software is at least partially responsible. The SCCM server should have aborted during the preflight checks when it realized it was going to take itself out in the process. The best thing this IT department can do is for the manager/director to keep the specific employe's name under wraps, stop shit from flowing down hill from above and move on. Nothing will benefit anyone if all of Emory treats the person responsible as if he deserves to pay for all the time lost in repairing the damage, he simply can't.
The hard lesson has been learned by everyone, nothing else will make anyone any better off.
Re:Sounds like IT incompetence (Score:4, Insightful)
Or he could just be an incompetent shit.
Don't get me wrong, I've made mistakes myself, perhaps not quite to the same level. Hopefully he is someone who can take a lesson but there are many who can't.
Only a good manager could tell the difference (Score:5, Insightful)
It sounds like the commenter above was teachable - he no doubt learned his lesson.
It also sounds like the company's owner knew he could learn this lesson. That's the mark of a great manager.
Whether the Emory staffer responsible for this mistake is teachable or not, I hope his boss can tell the difference. Some folks aren't teachable, some are. If the Emory boss is worth his paycheck, he should be able to tell.
Re:Only a good manager could tell the difference (Score:4, Insightful)
I used to work for a company called "Gteko". Don't bother looking them up - they were acquired several years ago. They sold bundled software (OEM) to a handful of companies, all of them huge. One of those was AOL. This is over a decade ago.
The incident in question took place after I left, so I don't know the specifics. The bottom line is, they screwed up a server deployment that affected the AOL front page for all AOL customers. After that was finally fixed, the company's CEO, expecting pretty much to be shown the door, walked into a meeting with several AOL high execs.
The meeting started with the following sentence:
"Let's see how we can make sure this never happens again"
Even when it's something less "close" to you than an employee, it is sometimes worth it to not terminate someone who made a mistake, even a serious one.
My current employer, Akamai, has a motto effectively saying: It's okay to screw up, so long as that screwup results in a procedure that will prevent anyone from making the same mistake again.
Shachar
Re:Sounds like IT incompetence (Score:5, Insightful)
Have you ever made a massive, expensive mistake?
Glances woefully down at wedding ring...
The person learned their lesson and will be extremely cautious in the future.
Thinks back on previous three weddings...
Re: (Score:2)
Have you ever made a massive, expensive mistake?
Glances woefully down at wedding ring...
+5 insightful
Re: (Score:3)
Fourth unhappy one? you're not making mistakes, you have a problem
Re:Sounds like IT incompetence (Score:5, Funny)
Fourth unhappy one? you're not making mistakes, you have a problem
Might as well face it, I'm addicted to love
Re: (Score:2)
marriage does not equal love, doesn't guarentee love, and carries many obligations and raises many issues having nothing to do with love
Re: (Score:2)
marriage does not equal love, doesn't guarentee love, and carries many obligations and raises many issues having nothing to do with love
you sound like that guy at the end of those life-affirming pharmaceutical commercials who reads all the horrible side effects super quick
Re: (Score:2)
past 50 and still in my first marriage, and things are still very good despite the occasional rocky times. but I know a lot of unhappy people who marry again and again, or feel "stuck" in miserable marriage. there are plenty of other ways to live life
Re:Sounds like IT incompetence (Score:4, Funny)
Fourth unhappy one? you're not making mistakes, you have a problem
Might as well face it, I'm addicted to love
What's love got to do with it?
It's a second hand emotion.
Re: (Score:2)
hahaha, fucking priceless :)
Re:Sounds like IT incompetence (Score:5, Funny)
Also known as:
Harrisberger's Fourth Law of the Lab:
Experience is directly proportional to the amount of equipment ruined.
Re: (Score:3)
Re: (Score:3)
I think it depends on the type of person you are, if you're usually a dutiful and reliable employee who made one mistake and it's a huge one that's different than your hotshot wonder boy who always does things the quick and dirty way and has caused minor outages and bugs before but gets away with it because the quick turnaround time is making him popular. Then I'd be a lot more inclined to say your reckless behavior finally blew up in everyone's face, there's the door. Admitting to your own screw-up is also
Re: (Score:2)
Assuming it was just a mistake and not malicious ...
Probably not. This shit happens, and that person who did it will never do something like this again. Have you ever made a massive, expensive mistake?
I have, I was 19 years old and cost my company nearly a million dollars due to a silly misconfiguration. After I discovered it, corrected the error and notified my boss, I spent most of the night throwing up. The next morning, after everyone in the company (only 15 people or so) knew what happened, and I walked through the halls on the way to the meeting with the owner and my boss, I thought I'd pass out. As I walked into the Owner's office I didn't even bother to sit down, expecting a fairly short conversation. I was asked to sit down while my boss had this very stern look on his face. So I did, cost them that much money, I can do what they ask.
The owner than proceeded to tell me the story of how, when working for a certain Germany car company doing CICS programming, he made a mistake that screwed up a production line and cost the company several million dollars. He knew exactly how I felt, and he knew that it would never happen again because I had already punished myself more than he possibly could.
If they fire the person who did this, they just wasted the whole event. The person learned their lesson and will be extremely cautious in the future. Firing them now just means someone else will get to reap the benefits of this experience, and thats pretty stupid.
People make mistakes, and in this case the software is at least partially responsible. The SCCM server should have aborted during the preflight checks when it realized it was going to take itself out in the process. The best thing this IT department can do is for the manager/director to keep the specific employe's name under wraps, stop shit from flowing down hill from above and move on. Nothing will benefit anyone if all of Emory treats the person responsible as if he deserves to pay for all the time lost in repairing the damage, he simply can't.
The hard lesson has been learned by everyone, nothing else will make anyone any better off.
While inspiring, the only reason you didn't get fired is, he did the same thing. He knew where you come from. 98% don't. I am betting that your boss had the termination papers in his desk already filled out and signed by him. However, since he was overruled by the owner you gained valuable insight. Be lucky. The IT person won't have that. He will be out on his/her ass because of that level of mistake. You are correct, someone else will get to reap the benefits of this experience, and thats pretty stupid. It
Re: (Score:2)
That is a nice story, and if true you got lucky that it was a small company and your boss probably knew your actual competency level.
In most places when stuff like this happens, your bosses' bosses' boss will want blood, and a nice firing will happen no matter what.
Protip: if anyone ever find themselves on the short end of this stick, don't grovel to keep your job. If possible, don't even discuss what happened. Remind them of your strengths, experience, what you can continue to contribute, and why they hir
Re: (Score:2)
From being on the receiving end of such begging it just makes you feel even less respect for the person doing the begging. If people are not aware of those things before a big mistake they are unlikely to want to hear about them afterwards.
Re:Sounds like IT incompetence (Score:5, Interesting)
SCCM is pretty good. It makes my desktop techs jobs significantly easier to deploy assets company wide. In this case, it sounds like someone pressed some buttons without being 100% clear as to what was going on. Unfortunate someone will not be working in IT ever again.
Or perhaps someone decided that having a testing environment for deployment packages was an unnecessary expense combined with personnel who aren't properly trained. Just think how much money they saved by eliminating training and a test environment!
Re: (Score:2)
Re: (Score:2)
if they have tenure then you can't get rid of them just like that and even then the logs may of been wiped out as well.
Re: (Score:2)
IMHO the actual incompetence is omitting a modal alert and just wiping the disk without asking the user first. Or should I say arrogance?
Re:Sounds like IT incompetence (Score:4, Insightful)
People make mistakes. Everybody makes them, everybody does it all the time, and they do it even when they should know better, when the consequences are high, and when they've received training specifically aimed at avoiding those particular mistakes.
Aviation, process and other industries know this by now, after many, many hard-earned lessons. They know you have to design your interfaces under the assumption that people will screw up, push the wrong button, or misread the situation. The general software industry, on the other hand, seems amazingly resilient against accepting this simple fact.
Surprisingly Infrequent (Score:5, Interesting)
I think the big surprise here is that this doesn't happen more often.
Consider how many corporations, universities, and such have huge PC deployments with automated updates. I've seen updates that drop all the PCs off the network, but I've never seen one where everything is wiped.
I'm also surprised that I haven't heard of malware that accidentally wiped a network of 100K or more machines when someone sent the wrong command.
Or maybe the news here is that it was in a more open environment where people hear about it. If a publicly traded company wiped a thousand PCs at its headquarters, you bet they would try to keep it quiet.
Re: (Score:2)
Maybe some picked all systems as the target for the this new image? But desktops and severs are groped together? may it was an all that is not used that much if at all. Now I can see some picking all desktops / laptops and trying to load the same image to all of them and having that fail due to hardware / driver issues. But servers as well?
Servers have a different OS and software load and they don't need the basic desktop apps like adobe stuff, MS office.
Re:Surprisingly Infrequent (Score:4, Insightful)
We use SCCM extensively at my office, and yes, it's entirely possible to tell it to reimage every single computer. You just need to target the deployment at "All Systems" and make it mandatory. My guess is that some admin picked the wrong collection, which is fairly easy to do in SCCM 2007 (2012 has Collection folders, which helps with that), and there's no warning messages -- just a summary of "this deployment is going to these devices, click Finish to do it." Of course, most other mass management tools assume that the admins know what they're doing, so they don't have much in the way of guard rails either.
One of the more obnoxious elements of SCCM is that there's no real way to recall a command you send out; clients pick up policy at periodic intervals, and without manual intervention, they'll just grab the policy and do what it says even if you kill the server in question. You can block deployments by taking down distribution points (if the clients can't grab content, they won't run the deployment), but you still have to be fairly quick about it to stop it.
What we do to prevent these sorts of disasters is implement process around the use of the ConfigMgr console and ensure only the people who know how to use it actually use it. To prevent an OS reimaging incident, our OS deployments go through a static set of collections by process and are always optional (requiring a manual touch, either at PXE boot or in the UI) except for a specific set of collections that are segregated in their own folder and have names and descriptions with scary words that make it clear what's going to happen. For instance, in our "Clean Reimage" folder, we have a collection that says, "Windows 7 Reimage (Clean, PXE, Forced)" with a description to the effect of, "*** A computer placed in this collection will be REIMAGED and LOSE ALL LOCAL DATA. Local state is NOT preserved or transferred. ***" If we were a larger IT organization, we'd probably use SCCM's role-based security to limit access to clean reimages to a specific group of people.
Re: (Score:2, Informative)
Pah, this is is small fry. If you want to see a proper clusterfuck check out what happened at the Dept for Work and Pensions in the UK in 2004/2005 (I forget re exact year). Somebody at EDS sent out a test update to 87,000 computers rather than the 200 or so it was supposed to gone to. They got the negation wrong in the logic and so the real network was targeted.
Just to really, really screw things up, the update partially failed so the machines were left in an inconsistent state and couldn't be used in any
Oh man (Score:3)
I bet the IT department is changing each other's diapers now! And updating their resumés....
Re:Oh man (Score:5, Funny)
In a résumé, "Watched in horror as images were accidentally deployed" becomes "Supervised the deployment of images on university-managed computers".
Re: (Score:2)
site-wide.
Not so sensational... (Score:3)
As someone who regularly provides consultation to IT staff, I know full well that there's plenty of 'administrators' that wade into waters they don't understand. We often encounter the aging IT staff member that's forced to interact with software they don't quite understand or we have the younger IT staff that impulsively click on what they don't understand, both occasionally leading a company to some manner of pandemonium level disaster. Or you simply have a dysfunctional IT department that doesn't communicate and, "oh, I'll just move this server into this container right here..." Just another day in IT.
Re: (Score:3)
Re:Not so sensational... (Score:4, Insightful)
That doesn't matter so much because things are changing at such a glacial speed. It may as well be 1999 for the small amount of 64 bit, multithreaded stuff that uses network capability well which is out there. If you defrosted a Sun sparc user from back then and put them on a Win8 machine they would be disappointed.
An...accident..? (Score:5, Insightful)
Knowing that people have been running various kinds of centralized update services, perhaps across multiple OSes, and spanning several years now, listening to a story about an update server literally going rogue and nuking everything attached to it, and then for the coup de grace, basically committing suicide at the end by reformatting itself, does not sound like an accident.
If it truly was, I'd hate to see what the hell purposeful intent looks like.
Re: (Score:2)
Systems with central administration has always been absolutely wide open to insider sabotage. Distributed systems can be made at least somewhat damage-limiting
As with others, I am amazed it doesn't happen either accidentally, or on purpose, far more than it does. You are basically one bit, or checkbox, away from it more-or-less all the time.
BTW, I note that the result was installing Windows 7. If it was doing that, does it mean they were running XP or Vista until just now?
Re: (Score:2)
BTW, I note that the result was installing Windows 7. If it was doing that, does it mean they were running XP or Vista until just now
Reimaging machines that already have win7 on them with a new win7 image is hardly an unusual thing to do. The intended target could have been an upgrade project or it could have just been a cleanup/update job keeping the same windows version.
I suspect the machines that got accidentally reimaged were running a mixture of stuff before. Thats how things usually end up in a uni environment (at the uni I left recently I was aware of machines running NT4, 2K, XP, Vista, win7 and win8 as well as osx and multiple f
Re:An...accident..? (Score:5, Funny)
Might be interesting to see how the Emory Board files this away.
Re: (Score:2)
Maybe the malware was deeply employed in all the install images, and now it is guaranteed that all the systems have it, even after re-deploying ;).
Re: (Score:2)
Knowing that people have been running various kinds of centralized update services, perhaps across multiple OSes, and spanning several years now, listening to a story about an update server literally going rogue and nuking everything attached to it, and then for the coup de grace, basically committing suicide at the end by reformatting itself, does not sound like an accident.
If it truly was, I'd hate to see what the hell purposeful intent looks like.
No that I think about it, you are right digital seppuku isn't an accident. This is something that I would hear on Coast to Coast AM under Illumanti or on Alex jones.
Re: (Score:2)
Knowing that people have been running various kinds of centralized update services, perhaps across multiple OSes, and spanning several years now, listening to a story about an update server literally going rogue and nuking everything attached to it, and then for the coup de grace, basically committing suicide at the end by reformatting itself, does not sound like an accident.
If it truly was, I'd hate to see what the hell purposeful intent looks like.
No that I think about it, you are right digital seppuku isn't an accident. This is something that I would hear on Coast to Coast AM under Illumanti or on Alex jones.
My point here was the focus on the amount of destruction that was caused by an "oops", not to shine a light on some batshit theory about uber-secret state-sponsored malware mass-injections.
Besides, we've already seen it's far easier to just compromise CAs anyway, and that's no batshit theory.
When update servers can be "weaponized" that easily against your own environment, it makes you wonder who really needs to be holding those keys, or perhaps why you even need to drive such a fancy update service.
Re: (Score:2)
Knowing that people have been running various kinds of centralized update services, perhaps across multiple OSes, and spanning several years now, listening to a story about an update server literally going rogue and nuking everything attached to it, and then for the coup de grace, basically committing suicide at the end by reformatting itself, does not sound like an accident.
If it truly was, I'd hate to see what the hell purposeful intent looks like.
No that I think about it, you are right digital seppuku isn't an accident. This is something that I would hear on Coast to Coast AM under Illumanti or on Alex jones.
My point here was the focus on the amount of destruction that was caused by an "oops", not to shine a light on some batshit theory about uber-secret state-sponsored malware mass-injections.
Besides, we've already seen it's far easier to just compromise CAs anyway, and that's no batshit theory.
When update servers can be "weaponized" that easily against your own environment, it makes you wonder who really needs to be holding those keys, or perhaps why you even need to drive such a fancy update service.
I agree 100%. It was mostly likely massive epic fail by someone who incompetence rivals those who built Healthcare dot org. Howvere, one CAN"T overlook the fact that people WILL call this a conspiracy by said agents because it fits the profile too easily.
Re:An...accident..? (Score:4, Informative)
This isn't the update server section of System Center (WSUS), it's the machine deployment system (Configuration Manager), and it can quite easily do this if left as-is out of the box with multiple technicians on it. And it can be done accidentally.
Here's the scenario as it likely happened.
We've had two near-misses with misconfigured collections and one hit with a different problem* which cannot have happened in this case. SCCM isn't the most intuitive user interface and if you're being pressured by users or trying to get out of the door for the weekend, you can stuff it up easily.
Our solution was to restrict access to the built-in collections and to build collections per computer lab which are presented as read-only to the technicians. And then gave them a day of lectures. It sort of works.
* The other problem was caused by image dumping with Ghost of an image that was sysprepped, but had the SCCM client still installed on the image. Because of that, several dozen PCs had clients with the same client ID, like the Windows GUID, but separate and not cleared by a sysprep. The technician later built a SCCM image and deployed it correctly to one PC in a personal collection. Unfortunately SCCM populated the deployment list based on the client ID of the PC in the list and hit quite a few overnight. Luckily a lot of the machines in the batch were off overnight. I don't think this is the case because it hit the server too and that would have received a new client install during the SCCM installation.
Re: (Score:2)
Trying to become famous by taking photos. Visit my homepage please.
Try harder: put a link in your sig.
Re: (Score:3)
intent would be a lot more subtle. An accident is just going to be big and obvious.
I guess that would depend on what the intent was.
In this particular case, if the intent was maximum damage and disruption, I've got a two word summary.
Nailed it.
Re: (Score:2)
This is semi-maximum damage and disruption. The users PC's would still work, albeit with no personal data. Given the way SCCM formats and dumps, there is a change of data recovery with any of the post-format recovery tools like EZRecovery, Recuva etc.
Max disruption would be to deploy a DoD-level hard disk wipe utility configured for 20 passes.
Re: (Score:2)
This is semi-maximum damage and disruption. The users PC's would still work, albeit with no personal data. Given the way SCCM formats and dumps, there is a change of data recovery with any of the post-format recovery tools like EZRecovery, Recuva etc.
Max disruption would be to deploy a DoD-level hard disk wipe utility configured for 20 passes.
That my friend, is a matter of perspective.
The average user staring at a freshly re-installed computer void of all personal data would not even remotely claim to see a difference.
The redness in their face and the volume of their voice as you try and claim otherwise would reaffirm that belief, especially if you have to tell them that the tape backups only captured the lost data on the servers...
Wrong OS (Score:5, Funny)
Backups (Score:4, Insightful)
Bad news most likely on this front. I have worked University IT, and I can guarantee they are going to have problems.
For one, no matter how many layers of backups you have, when you are working with a bunch of 90 year old academics, they will always find a way to miss every single one.
And more grievous, Universities tend to have important data that absolutely cannot be backed up in any normal way. Data that is legally obligated to stay on one specific computer in one specific room and never leave; under penalty of legal action.
Re: (Score:2)
Bad news most likely on this front. I have worked University IT, and I can guarantee they are going to have problems.
For one, no matter how many layers of backups you have, when you are working with a bunch of 90 year old academics, they will always find a way to miss every single one.
And more grievous, Universities tend to have important data that absolutely cannot be backed up in any normal way. Data that is legally obligated to stay on one specific computer in one specific room and never leave; under penalty of legal action.
That level of insanity is why I am laughing. The bold parts specifically. When you allow people who have no clue how a system works to legislate how it works, you get this.
Re: (Score:2)
Data that is legally obligated to stay on one specific computer in one specific room and never leave; under penalty of legal action.
That level of insanity is why I am laughing. The bold parts specifically. When you allow people who have no clue how a system works to legislate how it works, you get this.
And sometimes they understand and it's intentionally awkward. For example if you want to prohibit certain types of research instead of just legislating directly against it and appearing anti-science, just a
Re: (Score:2)
...Universities tend to have important data that absolutely cannot be backed up in any normal way. Data that is legally obligated to stay on one specific computer in one specific room and never leave; under penalty of legal action.
Hate this. Why keep the data where it can be encrypted and backed-up? Why keep it in a secure location as opposed to the Primary Investigator's (PI) office? Gah.
Re: (Score:2)
Privacy, even the IT admin cannot have access, and no system IT breach should leave it vulnerable.
This way, yes it is a pain in the ass and easy to lose, but for someone to leak it it requires someone to be in the room in person (and there is no copy paste all confidential reports to some thumb drive, as the confidential stuff is stored in different ways in different rooms all over campus).
Re: (Score:3)
... my professor's computer is not managed by the university, nor is mine. Our data would be OK.
I hope you verified this before posting. ;-)
Since a reformat and reinstall was done, the permissions involved were presumably handled at a lower level (BIOS?) than the installed OS. So it could easily have hit any Intel-based machines accessible via the network. Such low-level operations are rarely done by software that understands subtleties like ownership and organizational structures.
It might be interesting to know whether non-Windows and/or non-centrally-managed machines were affected by this ev
netboot or even an windows script that reformats t (Score:2)
netboot or even an windows script that reformat the system will run on system targeted systems. Also dells, HP, both AMD and Intel and have ways for windows apps to set bios settings and some places have netboot as the default setting in boot order.
centralized user data... (Score:2)
If at my workplace a computer breaks, gets stolen, catches fire, whatever, I fetch a new one from the basement, tell the PXE server to load the installation image. 15 minutes later, the user can resume their work.
Then again, it's probably much more complicated to achieve this with Windows.
Re: (Score:2)
The downside of centralisation is when things DO go wrong they go wrong in a big way. When your centralised data storage goes on the blink or your centralised imaging system decides to reimage every machine the whole company goes down.
Of course a halfway house can be the worst of both worlds......
Re: (Score:2)
The downside of centralisation is when things DO go wrong they go wrong in a big way.
Amen. It's not a bug - it's a big fat feature, very useful and very dangerous also.
Confucius probably say: He who think big gun always better than small gun, blow whole f***ing leg off.
Common at colleges for shared computers. (Score:3)
When I worked in the IT in my work study program, shared computers would be re-imaged often. We would usally re-image 300-400 computers at time. Often it was just some professor wanting certain program, it was just easier and safer just to wipe the machines. Malware was big problem at the time, sasser hit all of are computers, that sucked.
"Somehow"??? (Score:4, Insightful)
"Somehow" makes it sound mysterious and inexplicable. I'd be willing to bet that the truth is far less sensational. I could see a student tech assistant doing something like this on a dare, or a low-skilled admin just clicking OK one too many times, without actually reading the warning message.
Re: (Score:2)
Let's see a tech goes we are deploying the new windows 7 image to all systems right? get's a yes from some other tech / boss (thinking that that tech is thinking all desktops / laptops or all systems in group X) and picks ALL systems (that has the severs in it as well) but not all desktops / laptops all systems in group X.
Stay classy, Slashdot (Score:3)
Not only have they had to re-image all their PCs, you've now slashdotted their web server!
Now realize that with central update repositories, (Score:2)
It could happen to the whole planet.
And I have seen a lot of machines try to PXE boot by default. That has got to stop.
Backups - the happy ending (Score:2)
"Because the university ran nightly backups, they were able to restore all the PCs to the correct state later that afternoon."
If they had a centralized SCCM server, they should have a central backup server as well.
This is why .... (Score:2)
Re: (Score:2)
Re: (Score:3, Informative)
This is University. Based on my experience with University IT, there will be loads and loads of important data that you are legally obligated to NOT do that to. It cannot leave one specific room, in any form.
Normally, the computers are still contacted to the network and the Internet, but everyone using them must know NOT to copy any of these files off of C.
Re: (Score:3)
So there are laws which dictate which hard drives and/or appliances store data relative to the OS? They can still be in the same room if that's the concern but if there are laws that actually say "x, y, and z must be stored on the same partition as the operating system" then I say they get what they deserve and perhaps those laws need to be re-examined.
Re: (Score:2)
Even if you have a D: drive it's possible that the "recovery" wipes everything out or at least the partition table.
Re: (Score:3)
That would be silly. ;)
It would be as silly as... wiping all the computers of an entire university
Re: (Score:2)
People may keep big files local and the servers where also being formed and trying to install windows 7 as well.
Re: (Score:2)
In my company IT tests all upgrades on the isolated small network before deploying. You would imagine that this should be standard practice.
But there's no saying that this would have prevented the problem, so it's irrelevant. You can fatfinger just as easily in production as in test and staging.
And you can even introduce new problems if not careful. Like a script washing the test deployment environment, like clearing the test target IP by mask, and inadvertently leaving the network address in its place.
With no information about what the real cause was, advocating a cure is not very helpful. Even if it's best practice and something that shou
Re: (Score:2)
Sure but the test environment will almost never be absoloutely identical to the production one. So there is still room for unexpected admin screwups when deploying to the production system.
Re: (Score:3)
So it goes, so it has been, so it will be.
Ecclesiastes 1:9 and from Battlestar Galactica the new one. "All this has happened before and will happen again."