Cisco Patches 'Black Hat' IOS Flaw 66
thursnick writes "eWeek is reporting that Cisco has finally issued a comprehensive fix for a critical IOS vulnerability that set off a firestorm of controversy at the Black Hat Briefings earlier this year. The patches come more than three months after former ISS researcher Michael Lynn quit his job to present the first-ever example of exploit shellcode in Cisco IOS (Internetwork Operating System), a presentation that landed him in legal hot water. Cisco's advisory effectively confirmed Lynn's summer warning that the flaw could be exploited by remote attackers to execute arbitrary commands or cause a denial-of-service on compromised routers."
Wow, quick turnaround... (Score:3, Funny)
Re:Wow, quick turnaround... (Score:2)
Why not earlier? (Score:2, Insightful)
Re:Why not earlier? (Score:5, Informative)
If you read TFA, the bug involved system timers and how they were handled. Given that this probably affects most of the system functions, it's not surprising that it would take a while to make the changes and test it. Think about how long it took to fix the VM bugs in linux 2.4, this probably a change of similar magnitude.
Hold on a second here... (Score:2)
They fixed the VM bugs in 2.4?
Re:Why not earlier? (Score:1)
Cisco vs. Microsoft (Score:4, Funny)
Re:Cisco vs. Microsoft (Score:3, Funny)
From: <billgatus.of.borg>
To: <ceo.of.cisco>
"Johnny, you're doin' a heck of a job!"
Black hats, white hats, Red Hats (Score:1, Funny)
Re: (Score:1)
hrmmm (Score:1)
patching ciscos... (Score:2, Informative)
Or do we have to manually evaluate lengthy decision diagrams, check memory requirements, prove that we have legally bought the affected hardware and software, and hope that the monolythic IOS image will not introduce bugs into other areas that are being patched by this fix?
Re:patching ciscos... (Score:2)
(Not that non-monolithic systems are necessarily exempt from the patch breaking other systems)
However (while off topic), it should be noted that 12.2XR (7600 only, today, but where else are you going to see this level of change) is no longer monolithic. It's a HUGE change in the architecture brought about to address just the type of issues discussed.
What ever happened... (Score:5, Interesting)
Re:What ever happened... (Score:5, Informative)
I know that he has a new job and I while I obviously can't speak for him, I got the impression that he felt as if he did his duty the security community. As an amateur member of that community, I'd thought that he put principle before pay and deserves our respect.
Simon.
Re:What ever happened... (Score:5, Informative)
Re:What ever happened... (Score:4, Insightful)
The coverup is almost always worse than the crime in these kinds of things. Companies that aren't up-front and honest (trying to protect their reputation) end up trashing their reps. Cisco just created an anecdote for the next time a customer or regulator wants to take a deep, careful look at their security. We can't just take their word for it, and if I were buying routers right now, I'd be much more inclined to look at Juniper than Cisco, even though previously I wouldn't have even considered them.
It's not magic pixie dust, but making the effort to bring hard-core ethics onstaff is important to me.
Seems CISCO should be in hot water.. (Score:2)
Re:Seems CISCO should be in hot water.. (Score:1)
Initial problem already fixed? (Score:1)
Re:Initial problem already fixed? (Score:1)
Everyone always bashes MS for releasing patches that simply prevent a working/known exploit w/o fixing the core issue. Cisco has done both now and even though I think they handled this very poorly on both the PR front, and with their behaviour towards Lynn, I do applaud them for taking the two step a
The question is....... (Score:4, Interesting)
Re:The question is....... (Score:5, Informative)
This code has been out for a few months now, and many select beta sites have been testing it in production environments. The first few iterations had some serious (crash and reboot every few hours) problems, but it (12.2.15T1thru17) has been in production use on several edge routers for a month with no noticable problems. Cisco didn't just patch the one 'sploit published, they categorised the class of exploits and went about fixing many different possible attack vectors or watching for suspicious behaviour that could indicate a compromised system. That is what took several months even before Michael's talk, and its been in testing (and re-patching and recursion testing) since then. The announcement today is because they are confident their fix is solid, but anyone staying at the bleeding edge of IOS releases has been using it since at least June.
I'd say its solid, but I'm not rolling out the latest version on everything until others add some real world stress testing. I'm sure there will be several more newly introduced bugs uncovered in the new few months, and the timer checks usually result in a panic reload, not optimal for stable systems with SLAs and big money riding on them.
I'm also not in a rush to roll this out, because for the moment there are no known exploits running around. Maybe Effugas or some of the IOS engineers (I know you read
the AC
Re:The question is....... (Score:1)
Re:The question is....... (Score:2)
Great (Score:3, Funny)
Boy oh boy (Score:3, Funny)
Re:Boy oh boy (Score:1)
copy flash tftp
copy run tftp
copy tftp flash
reload
not too bad..
Re:Boy oh boy (Score:1)
Remove fan grill
Insert penis
Pray that the networking gods accept the sacrifice placed before them
Re:Boy oh boy (Score:2)
Two scary bits" Completely Compromised (Score:3, Interesting)
"In many cases, a heap-based overflow in Cisco IOS will simply corrupt system memory and trigger a system reload when detected by the "Check Heaps" process, which constantly monitors for such memory corruption."
Is anyone else bothered that Cisco figures heap corruption is common enough that a process is running full time on production routers looking for it? I suppose you could view this as proactive, but obviously the process can only look for nonmalicious corruption, and is only statistically likely to find corruption before it causes errors according to how much CPU you give it.
"In some cases it is possible to overwrite areas of system memory and execute arbitrary code from those locations. In the event of successful remote code execution, device integrity will have been completely compromised,"
Think about it. Once an exploit is executed against your router, reloading your firmware isn't an option, because that's a function of your firmware, which could be corrupted. Unlike a computer OS virus, which can be circumvented by rebooting and taking control before the corrupted OS does, there's no way to preempt the corruption here. For total peace of mind, you'd either have to replace the (probably not socketed) flash chips, or take the whole unit out back and burn it. Am I wrong? Of course, that's not going to be Cisco's recommended solution.
Re:Two scary bits" Completely Compromised (Score:2)
I'm bothered. In fact, this reminds me of how DJB's software almost always users the supervise daemon to ensure your process is running. It keeps track of all of djb's software, and you can run it with most other daemons.
What happened to writing good software? Why should you have a daemon check for corruption? Why should you have a daemon that checks to see if other daemons die? Wait! Wasn't the author of the daemon-that-might-die so
Re:Two scary bits" Completely Compromised (Score:2)
Process external input.
Exceptions are exceptional.
Expect the unexpected.
Re:Two scary bits" Completely Compromised (Score:4, Informative)
As for reloading firmware, I don't think you understand Cisco stuff. There is a mini-firmware burned into ROM on all the Routers & Switches...it's called ROMMON mode on the ones that immediately come to mind. If your device firmware is totally thrashed (by a worm, by some damn fool tftp'ing up an image for the wrong router type, etc) you'd just use ROMMON mode to re-load a good image. Now, the real problem is that a worm could trash your flash storage.
In that case, unless you've got one of the expensive boxes with removable flash cards, you've now got a very expensive paperweight.
Re:Two scary bits" Completely Compromised (Score:1, Interesting)
Cisco doing heap checking is a mark of a reasonable system doing checks on itself.
It's a mark of a *bad* system. Why? Because 1) it means they believe they haven't properly written their software and, more importantly, 2) it doesn't guarantee you anything except "the heap was consistent up to 29 seconds ago". Who cares? I need the heap to be consistent *all* the time.
The best thing is to just write the code correctly, the next best thing is to place some kind of "barrier" (at the hardware level? Who
Re:Two scary bits" Completely Compromised (Score:2)
Well most cisco routers have socketed and/or slot based flash. The slot based ones have these realy cute write protect switches on the end.
"First-ever exploit" (Score:1, Informative)
It was not the first-ever example of exploit shellcode in IOS, Phenoe [phenoelit.de]
Cisco not aware of an exploit? (Score:2)
Right.
Am I affected? (Score:2)
My router has version 12.4(2)T1, is it affected? The advisory says that all version are affected, but it seems to propose version 12.4(2)T1 as a fix.
Could someone shed some light one this?
Re:Am I affected? (Score:3, Funny)
Re:Am I affected? (Score:2, Informative)
Upgrade Procedure (Score:2)
At least that's how it worked for me this morning. The entire process took less than 2 hours from initial email to downloading the updated version of IOS.
BTW: be sure to quote the advisory URL in all of your emails to Cisco.
Re:Great news (Score:2, Informative)
I'm Michael Lynn, so I know a thing or two about what went on...I DID NOT release any bug details, I DID work with the vendor, the bug in question was patched months before I went on stage as a result of my working with PSIRT, and when I went on stage I didn't disclose any details about any bug...all I did was prove it was possible to exploit bugs on IOS...
If you don't belie
Re:Great news (Score:2)
Thanks for going public. I hope you get a chance to read this, but as far as I'm concerned and will tell anyone who asks that you did everything right and it was Cisco who screwed themselves.
I've heard few people say differently, either.