Myself I ended up at work 20 hours on Monday this week patching servers. Given that we have about 500 servers in our environment with one person doing the patching this wasn't so bad.
We ended up with a lot of problem because of this worm... less because it actually caused problems with the machines but more because we could see machines constantly trying to infect one another. It wasn't pretty. Our workstations were most at risk, being the largest installed base but also running Windows 2000 SP3 (not SP4 unfortunately). No patch has been generally released for SP3 WS's, but a custom patch IS available from Microsoft if you request it. Due to other factors in play, we have elected to upgrade to SP4 and install the appropriate hotfixes. This is not going to be pretty over about 10,000 workstations.
See, what some people miss when they say that any infection may be due to bad administration is simply that we're dealing with huge numbers of machines, both servers and workstations that are potentially vulnerable. Due to application compatibility and tested standardized platforms we often don't even get the option to keep stuff up to date. The only reason we even have Windows 2003 servers in place today is because we forced the issue with our Corporate guys when we implemented Active Directory; we informed them that we had a need for functionality not provided by Windows 2000 AD (which was true). There is a project currently under way to test Windows XP for rollout, but honestly chances are that Vista will be shipping by the time we even reach 50% rollout mark.
So, why the rant? Well, it must be understood that jumping on the latest patches is not always an option in the corporate environment. Also, jumping on the operating system bandwagon is rarely an option because there's a lot of regression testing that has to be done. Hell, there are some instances where we're having to push the application vendors to support Windows 2003 Servers in our Citrix environment because they've never tested it. Welcome to the realities of Corporate IT.
Are there solutions? Sure! However, none of them are acceptable to most corporations. Linux is not an option, neither is OSX. In both cases we come back to the legacy support issue. Citrix to share the applications? Great... but you're only redirecting the problem to the server farms, not eliminating it. Real world Corporate IT is not as black and white as people would like it to be, myself included.
This virus gained traction because most corporations work this way. It wasn't helped by the fact that McAfee and Symantec both waited two days after the virus was discovered to release a signature update that recognized it.
One positive thing though; this virus is forcing the management to finally listen to my department's complaints that we need to be more proactive about patch management, and this time stuff might get done. We've got a long way to go, but this should be the start of something better.
I agree with your assessment of the tribulations of large network administration. You might have 500 servers and thousands of workstations to manage, but how many gateways to public networks do you have? Substantially less, I'd wager. Would not proper firewalling have prevented this worm from entering the network in the first place? What about DHCP configuration that moves mobile/unknown hosts to an untrusted network, perhaps with carefully filtered VPN only access?
In this case, no. Although we can't pinpoint it, it looks as though this worm actually came in on an infected laptop. There's almost nothing that can realistically be done to prevent this unless we also want to force everyone to use desktops. I know a lot of managers (and IT people... myself included) who often work from coffee shops on wireless connections when we need to. It people like myself can be expected to be conscientious about using at least a software firewall; managers and project managers? Well
I centainly agree about expectations of the non-IT staff to be conscientious about those things.:)
That is why I don't trust them. My DHCP server serves up a different pool of IPs to mobile computers, which are firewalled from doing anything except connecting to the terminal server.
Granted, this is not practical for everyone, but the concept of "least access required to get the job done" can be utilized in many other ways.
Unfortunately, again comes the Corporate politics. We tried to implement something like this, but all it took was one manager with a valid beef and a loud enough voice and all of a sudden we found ourselves back to basics. The problem with these kind of "ideal world" configurations is that there's nearly always one application that is overlooked. With almost 300 "custom applications" in our environment today THAT WE KNOW ABOUT (and probably twice that many we don't), hitting the nail on the head first time i
It might be worth looking at the Network Quarantine service in WS2003. I think WS R2 might have it available for the LAN side as well, which would enable you to do a quarantine and inspection before people with laptops can run amok through the network.
Software firewalls can work. A company I work with has over 10k laptops in use, and nearly all of them run a standard firewall package. It has centralized logging, so we can tell when someone disables it and/or uninstalls it. Those users are warned once and then walked out the door if it happens again, even managers. Patch management is handled automatically, so when a user logs on, the patch is pushed to them. If the firewalls are configured intelligently (i.e. absolutely NO MS networking allowed when in an untrusted network), patches are maintained, and antivirus software is in place, the virus problem gets much more managable.
Add to that an IDS that has provisions to automatically identify propegating worms inside the company, interfaces with the trouble-ticketing system, and a process through which access control lists are applied to the appropriate routers within 15 minutes, and you have a method for dealing with viruses quickly and without a bunch of manpower. These days, a bad virus outbreak for me is 2-3 computers, and we've got well over 40k end users.
We ended up with a lot of problem because of this worm... less because it actually caused problems with the machines but more because we could see machines constantly trying to infect one another.
Next time you should try blocking the botnet controllers at the network perimeter. Usually, this simple measure significantly reduces propagation on your internal network, even if you don't have internal compartmentalization.
I ended up at work 20 hours on Monday this week patching [...] about 500 servers
I'm honestly, not-trolling-ly curious what the procedure is for rolling out a patch to 'n' Windows machines, where 'n' is a large number like 500.
In the UNIX world, patching 5, 50, 500, or 5000 machines would be a piece of cake due to all the infrastructure tools useful for automation (bash, perl, ssh, rpm, patch, tar, etc.). Are there useful equivalents in the Windows world, or are you manually click-click-clicking on 500 mach
Oh no, the actual patching method was pretty simple, automated and realistically only ate up a total of about 4 or 5 of those hours. The problems came when it came to controlled reboots, reboot schedules, application and server interdependencies and so forth. Also, the politics of dealing with servers in remote locations and having to call on-call staff in the middle of the night to power-cycle a box because McAfee hung the server on shutdown. That's what causes time... and is common across platforms.
Which then brings up the problem of application compatibility. As I mentioned in one of my other posts (though not specifically), many of our "custom" or at least "not-off-the-shelf" applications are only certified for Windows 2000 SP3, and have either never been tested or never certified on anything newer. This leads to the problem of the vendor leading the customer down a dark and dangerous path, but unfortunately corporate politics plays too much into this. We aren't allowed to run un-certified applicatio
The other issue here is that the PnP fix was originally evaluated as a "medium" risk patch, not a Critical path. In our organization, critical patches get testing priority over lower risk issues.
Depends a lot on your point of view (Score:5, Interesting)
We ended up with a lot of problem because of this worm... less because it actually caused problems with the machines but more because we could see machines constantly trying to infect one another. It wasn't pretty. Our workstations were most at risk, being the largest installed base but also running Windows 2000 SP3 (not SP4 unfortunately). No patch has been generally released for SP3 WS's, but a custom patch IS available from Microsoft if you request it. Due to other factors in play, we have elected to upgrade to SP4 and install the appropriate hotfixes. This is not going to be pretty over about 10,000 workstations.
See, what some people miss when they say that any infection may be due to bad administration is simply that we're dealing with huge numbers of machines, both servers and workstations that are potentially vulnerable. Due to application compatibility and tested standardized platforms we often don't even get the option to keep stuff up to date. The only reason we even have Windows 2003 servers in place today is because we forced the issue with our Corporate guys when we implemented Active Directory; we informed them that we had a need for functionality not provided by Windows 2000 AD (which was true). There is a project currently under way to test Windows XP for rollout, but honestly chances are that Vista will be shipping by the time we even reach 50% rollout mark.
So, why the rant? Well, it must be understood that jumping on the latest patches is not always an option in the corporate environment. Also, jumping on the operating system bandwagon is rarely an option because there's a lot of regression testing that has to be done. Hell, there are some instances where we're having to push the application vendors to support Windows 2003 Servers in our Citrix environment because they've never tested it. Welcome to the realities of Corporate IT.
Are there solutions? Sure! However, none of them are acceptable to most corporations. Linux is not an option, neither is OSX. In both cases we come back to the legacy support issue. Citrix to share the applications? Great... but you're only redirecting the problem to the server farms, not eliminating it. Real world Corporate IT is not as black and white as people would like it to be, myself included.
This virus gained traction because most corporations work this way. It wasn't helped by the fact that McAfee and Symantec both waited two days after the virus was discovered to release a signature update that recognized it.
One positive thing though; this virus is forcing the management to finally listen to my department's complaints that we need to be more proactive about patch management, and this time stuff might get done. We've got a long way to go, but this should be the start of something better.
Re:Depends a lot on your point of view (Score:2, Interesting)
You might have 500 servers and thousands of workstations to manage, but how many gateways to public networks do you have? Substantially less, I'd wager. Would not proper firewalling have prevented this worm from entering the network in the first place? What about DHCP configuration that moves mobile/unknown hosts to an untrusted network, perhaps with carefully filtered VPN only access?
Simple to manage steps can certainly be t
Re:Depends a lot on your point of view (Score:3, Informative)
Re:Depends a lot on your point of view (Score:1)
That is why I don't trust them. My DHCP server serves up a different pool of IPs to mobile computers, which are firewalled from doing anything except connecting to the terminal server.
Granted, this is not practical for everyone, but the concept of "least access required to get the job done" can be utilized in many other ways.
Re:Depends a lot on your point of view (Score:2)
The problem with these kind of "ideal world" configurations is that there's nearly always one application that is overlooked. With almost 300 "custom applications" in our environment today THAT WE KNOW ABOUT (and probably twice that many we don't), hitting the nail on the head first time i
Re:Depends a lot on your point of view (Score:2)
These days, I get to be idealistic because the current administration values security over functionality, so to speak.
Re:Depends a lot on your point of view (Score:2)
Re:Depends a lot on your point of view (Score:4, Informative)
Re:Depends a lot on your point of view (Score:2)
Next time you should try blocking the botnet controllers at the network perimeter. Usually, this simple measure significantly reduces propagation on your internal network, even if you don't have internal compartmentalization.
Re:Depends a lot on your point of view (Score:2)
I'm honestly, not-trolling-ly curious what the procedure is for rolling out a patch to 'n' Windows machines, where 'n' is a large number like 500.
In the UNIX world, patching 5, 50, 500, or 5000 machines would be a piece of cake due to all the infrastructure tools useful for automation (bash, perl, ssh, rpm, patch, tar, etc.). Are there useful equivalents in the Windows world, or are you manually click-click-clicking on 500 mach
Re:Depends a lot on your point of view (Score:4, Interesting)
Re:Depends a lot on your point of view (Score:2)
Re:Depends a lot on your point of view (Score:2)
Re:Depends a lot on your point of view (Score:2)
Well, it must be understood that jumping on the latest patches is not always an option in the corporate environment.
Waiting almost 2 years to deploy Win2K SP4 is not exactly "jumping".
Re:Depends a lot on your point of view (Score:3, Interesting)
We aren't allowed to run un-certified applicatio
Re:Depends a lot on your point of view (Score:2)
Re:Depends a lot on your point of view (Score:1)