Debian Bug Leaves Private SSL/SSH Keys Guessable 670
SecurityBob writes "Debian package maintainers tend to very often modify the source code of the package they are maintaining so that it better fits into the distribution itself. However, most of the time, their changes are not sent back to upstream for validation, which might cause some tension between upstream developers and Debian packagers. Today, a critical security advisory has been released: a Debian packager modified the source code of OpenSSL back in 2006 so as to remove the seeding of OpenSSL random number generator, which in turns makes cryptographic key material generated on a Debian system guessable. The solution? Upgrade OpenSSL and re-generate all your SSH and SSL keys. This problem not only affects Debian, but also all its derivatives, such as Ubuntu." Reader RichiH also points to Debian's announcement and Ubuntu's announcement.
stupid stupid stupid (Score:5, Insightful)
did I mention stupid?
this is how some of the old video games were "broken" despite using "random" numbers, the seed was always the same... leading to the same sequence of events
The big question is.. (Score:5, Insightful)
But seriously, removing the code that seeds a random number generator? I can hardly imagine making such a change by accident. I may just lack a sufficiently colorful imagination, though.
(or, before resorting to conspiracy theories, we should probably ask ourselves first, "can this possibly be explained by simple stupidity?"
Re:stupid stupid stupid (Score:5, Insightful)
Who did this? You don't remove the seeding... stupid
That's what I was thinking too. What kind of idiot makes that sort of change?
Re:It will be fixed (Score:5, Insightful)
Ubuntu Gutsy already updated... (Score:3, Insightful)
OSS, only as good as the last developer? (Score:5, Insightful)
But the point here is that the freedom that OSS gives you does require you to trust the whole distribution chain. In this case there was an added muppet who did something they shouldn't have thus rendering everything downstream insecure. OSS is great but it required great developers, given that it has take well over a year to get the advisory out it shows that the many eyes piece didn't work here, mainly because the eyes were looking at the original source not the botched packaging job.
The "easy to use" Linux box in the house uses Ubuntu and has this issue and like many people I didn't even think to check that the OpenSSL wasn't the REAL OpenSSL it was OpenSSL with muppet extensions. Maybe there needs to be some form of extension that warns that a package has been modified from its original source code and that the modification was done by "K. Frog" so you can determine whether to trust that package or look back to the source.
Or some sort of voting system on contributors (how very Web 2.0) so you can see how the people who touched your package were rated with the biggest weighting being given to the last person through the code (hand edited by Linus = 5 stars, hand edited by James Gosling = 5 stars, hand edited by the bloke who wrote clippy = 2 stars, hand edited by a bloke who removed a seed generator = 0 stars).
Having the code is great, but this makes me want to know much more about who last edited that code.
Re:Many eyes make all bugs shallow (Score:2, Insightful)
Re:It will be fixed (Score:5, Insightful)
Debian people screwed up. This leaves a huge distaste in my mouth for Debian (and Ubuntu).
Re:It will be fixed (Score:5, Insightful)
note: When I have to run Linux instead of a BSD it's Debian and/or Kubuntu all the way since the benefits outweigh the negatives, but it's still an annoying habit of theirs.
Re:You stupid god damned open sourcers (Score:5, Insightful)
Yeah, like all those times when MS cut checks for all their customers whose computers were compromised! Oh, wait...
Re:The big question is.. (Score:2, Insightful)
Re:Of course... (Score:5, Insightful)
... and regenerate all the keys, yes? It isn't quite as simple as you suggest...
"All OpenSSH and X.509 keys generated on such systems must be considered untrustworthy, regardless of the system on which they are used, even after the update has been applied."
who was it? (Score:1, Insightful)
I cannot believe that such a change would have gone by unnoticed if it had been properly documented, and I cannot believe anyone given the trust of the OpenSSL distribution would make such a mistake.
This was an intentional act of sabotage.
Re:It will be fixed (Score:5, Insightful)
If someone stole your crypto keys and has had them for a year...
How thoroughly might they have compromised your system by now?
How many passwords might they have stolen that you use on other systems?
What else might they have done that will give them access in the future even after you fix this?
Just regenerate your keys and no problem? The problem that guessable keys are generated will undoubtably be fixed asap, if not already. The problem that this has been the case for the last year will not be, and is a big worry.
Re:OSS, only as good as the last developer? (Score:5, Insightful)
No software is perfect. when F/OSS screws up everything including the exact versions of the software where the bug began, until it is fixed is known. You know what/where/when/how, and most of the time why it happened.
With closed source software your considered lucky if you get a patch in a timely fashion.
Personally i would rather know what happened and when too.
A great filter (Score:5, Insightful)
Re:The big question is.. (Score:2, Insightful)
If the memory was initialized, then it wouldn't be an issue for valgrind, and the seed would still be whatever the memory had been initialized to.
Re:To non-IT people (Score:5, Insightful)
Re:OSS, only as good as the last developer? (Score:4, Insightful)
The issue is a bit more complicated... (Score:2, Insightful)
But the fact that changing the use of NON-INITIALISED memory causes most of the seed of a RNG to go away really is worrisome... unless that memory is actually initialized with a seed BUT in a way Valgrind and others can't track. Which means it is sloppy coding on OpenSSL.
You cannot trust unitialized memory, PERIOD. And it will take looking deep into OpenSSL code for me to tell if it was intialized in a way valgrind couldn't see, or that OpenSSL was working due to just plain blind luck (or lack of an attacker).
Either way, upstream prepared the trap, and a debian maintainer sprung it.
Why do YOU think we are not calling for blood on the guy who did the change? Because code that breaks like that is a travesty by itself.
Re:It will be fixed (Score:5, Insightful)
Re:The big question is.. (Score:2, Insightful)
On the other hand it is possible that the Debian maintainer didn't understand what he was doing, and his changes did more than remove that read into uninitialized memory. In either case it I can't imagine why someone would modify a RNG or any other cryptographic code without checking with the original author first before distributing it. That is just unacceptable.
Don't underestimate corporate arse-covering (Score:5, Insightful)
I, on the other hand, strongly suspect that any similar mistake at a major software corporation would in all probability be quietly ignored, if it were even noticed at all -- and if it were instead deemed enough of a public relations risk to warrant dealing with, the company would likely just silently push an update to correct the problem for future users, leaving anyone using extant keys with their arses hanging in the breeze.
But maybe I'm just being overly cynical. :-\
Cheers,
Re:interdiff (Score:2, Insightful)
Re:It will be fixed (Score:2, Insightful)
1) They shouldn't have messed with the NON-BROKEN Source (which every other distro uses), without testing it better.
2) I use whatever works, currently I have Ubuntu on my kids computers, Windows and SUSE and Mac OS and BSD at work. I'm more of a utilitarian; computers are nothing more than tools. Right tool for the job, makes all the difference in the world.
3) Reputations are hard to come by. Good ones are hard to maintain. I'm sorry for Debian, because they lost a tad bit of credibility on this one, in my book.
You are funny (Score:3, Insightful)
Bug found & fixed -- "many eyes" seems to work (Score:4, Insightful)
First off, we were not told any such thing -- or at least I've never run across any such claims myself. Secondly, this would actually seem to substantiate the many-eyes theory, as the bug was found and summarily corrected, rather than never found, or found and swept under the rug, or found first by black-hats and exploited, all of which seem quite common in the proprietary software world.
The many-eyes model does not guarantee zero bugs, nor does it guarantee speedy bug hunting. However, it would seem to guarantee that bugs will be found eventually, and also that they will be dealt with in some productive way once they are found. This model aims for the opposite of security through obscurity, that hoary old chestnut of proprietary development, by instead ensuring security by knowing exactly what things do. That also means that folks need to go through all changes -- i.e., many eyes actually need to go over the code. Open source ensures that this is possible, but actual people still need to put in the time and effort.
Cheers,
Re:Don't underestimate corporate arse-covering (Score:2, Insightful)
I think you are completely correct and that this is exactly what would happen in most cases with software that came out of a corporation. I don't think you're being cynical at all, just realistic.
Re:OSS, only as good as the last developer? (Score:4, Insightful)
Is it good enough now? (Score:3, Insightful)
Has someone actually checked the random output for cryptographic soundness?
The proper source of cryptographic-quality random bits in Linux is "/dev/random". Reading uninitialized memory is not a good source of entropy - it might be initialized to some constant value, especially if you're compiling with a debug allocator or running under a virtual machine monitor.
OpenSSL makes substantial efforts to get a good random starter. [openssl.org] Unless someone did something so stupid that OpenSSL didn't use /dev/random, it should still work. OpenSSL is supposed to have a check for bad random seeds. Was that bypassed, or doesn't it work, or what?
Obtaining a good source of randomness is hard. Computers are rather deterministic. Historically, there have been major failures in this area. See "Venona" [wikipedia.org] where the USSR was generating "one-time pads" by having people type random digits on typewriters. Arlington Hall, NSA's predecessor, cracked that. Humans aren't random enough. True random number generation requires special hardware [fourmilab.ch], like a noise diode or a radiation source.
"Any one who considers arithmetical methods of producing random digits is, of course, in a state of sin." -- John von Neumann
What does "guessable" mean here? (Score:3, Insightful)
Does that mean someone can now generate a "master-key" and ssh into root@any-debian-box that allows public key authentication?
What does a realistic attack scenario look like?
Re:It will be fixed (Score:4, Insightful)
That's not to say that upstream providers are unduly arrogant either -- if you're happy with the way your build/install system works, why would you want to maintain patches for some other system that you don't even use? It's extra code to keep up, and requires more testing on more platforms, and it it doesn't make your core package any better.
Re:stupid stupid stupid (Score:2, Insightful)
That's not a case of seeding too often, that's a case of using a crappy seed
Re:The big question is.. (Score:4, Insightful)
It isn't easy to find where the function they changed is actually called... it gets added to an array of function pointers and then a static variable is set to it, and then there are various macros that can obtain it. I can see how valgrind could be confused, but in any case there is a function that takes a buffer and adds it as randomness and they nuked it. They totally screwed up because they assumed that valgrind was perfect at figuring out what memory was initialized.
There is no simple excuse like 'it was uninitialized wanyway', the debian openssl team just blew it, big time.
Re:OSS, only as good as the last developer? (Score:3, Insightful)
Agreed.
The parent asked how can we be sure we're using the real OpenSSL? Simple, just use a distro that maintains a policy of minimal interference with upstream code. Slackware comes to mind, and not much else unfortunately.
Any distro that's liberal about patching upstream code is liable to suffer from security issues, as just exemplified by this story, and also to reduce the value of bug reports to the upstream projects, since they have to go through an extra triage to make sure the bug isn't specific to that distro.
Re:A great filter (Score:5, Insightful)
This is a big deal. Maybe it doesn't affect you very much. It doesn't affect me at all, I've never run these distros. But you can bet right now that there are a lot of heavy Linux users out there going through a lot of trouble. This is going to be a bonanza for certificate authorities as everyone who generated SSL keys with these distros is going to need to purchase a new cert.
Re:Surely this is not the only source of entropy! (Score:3, Insightful)
Re:It will be fixed (Score:4, Insightful)
Most users do not generate nor use ssh/ssl keys or certificates.
The real questions are
Re:How Frakin stupid can you be? (Score:3, Insightful)
One of the OpenSSL developers [links.org] (or at least, that's what I infer) puts it in even more general terms:
However, I can't help wondering if some fault may arguably lie with the OpenSSL coders. I mean... by his own admission:
(Emphasis mine)
So, since it's unusually doing something that looks an awful lot like a Cardinal Sin, did this block of code include a prominent:
NB: Yes, we are reading unitialised memory
This is deliberate, NOT A BUG!
<explanation here>
*/
I mean, if you're going to write code that basically looks like a duck, walks like a duck and quacks like a duck, and which you know is going to head downstream toward a huge bunch of duck-hunters, it's really a good idea to add a big visible note saying THIS IS NOT A DUCK.
Re:2 years? (Score:4, Insightful)
If 99.9% of client code works, then an obscure bug in BSD directory traversal code is pretty unimportant. It won't affect most people, so it won't be found by most people.
This kind of bug, on the other hand, severely harms the security of everyone using this distro. It's tricky because it's hard to tell the difference between good crypto and crypto which has a critical flaw at the beginning which makes it insecure, which is why it lasted for so long before being found. But still, one would think that crypto software would get more attention and that someone would have noticed that every affected machine is spitting out nearly identical keys.
Re:stupid stupid stupid (Score:5, Insightful)
Lesson#2: If you write code that deliberately does weird things like wanting to read unitialised memory, PUT A COMMENT so that people other than you have a fighting chance with your code.
Re:Surely this is not the only source of entropy! (Score:3, Insightful)
Yes, I get that, you're just restating what I quoted in slightly different words. My point is that it looks to me that at best it doesn't matter because it's not a good source of randomness.
They then call a randomisation routine. The broken patch commented out both calls.
Both calls were to the same routine, MD_Update, that added part of the uninitialized buffer to the message digest being built in two different places.
Don't mess with code you don't understand, especially when it is so important.
I'm not messing with it, I'm questioning it. If that code is important, then I'd like an explanation of why they think this is a good sort of randomness to depend on.
If they don't want to use
Heck, you can do a lot better by deliberately taking advantage of known variations in the environment: they'd do better hashing
Re:It will be fixed (Score:5, Insightful)
Re:Why open source doesn't work for business (Score:4, Insightful)
Or a bit like asking Microsoft to accept responsibility for the copy of Windows that you paid for.
Look through the Windows license agreement some time and see how many times phrases like "THERE IS NO WARRANTY OR CONDITION OF ANY KIND", "YOU ARE NOT ENTITLED TO ANY DAMAGES" and "IN NO EVENT SHALL MICROSOFT OR ITS SUPPLIERS BE LIABLE" show up.
The only difference with Microsoft is that if Windows breaks you don't get to keep both pieces.
Re:Surely this is not the only source of entropy! (Score:4, Insightful)
Well yeah, especially because a byte of "uninitialized" memory, like all memory regardless of how many times it has been initialized, is much more likely to contain the value zero than any other. That'd be like using a 6-sided die as a source of randomness when the '1' side was ten times the area of the '6' side.
However as mentioned, that's not the only or main source of randomness, and getting rid of that randomness was not the bug. It was getting rid of other sources of randomness in the process, because they -resembled- the function that used uninitialized memory.
Re:It will be fixed (Score:5, Insightful)
No, he posted a question openssl-dev, which is a mailing list for people writing software that uses OpenSSL. This OpenSSL developer doesn't read it [links.org] and that's similar with most open source project - developers often don't read mailing lists for end users.
What's more, that mail doesn't contain a patch. It contains a misleading question with two lines posted in isolation. An actual patch, submitted for an actual code review, would almost certainly have revealed the problem via context.
You don't change crypto code of all things based on an idle question on a mailing list populated mostly by users. What's next, changing the kernel scheduler based on a conversation in #kernel-newbies?
Re:It will be fixed (Score:3, Insightful)
Once again, shoddy code leads to errors (Score:3, Insightful)
Given that the code is so shoddily written that neither of the functions has a comment to say what it's supposed to do, I think I can forgive him for that.
Re:It will be fixed (Score:3, Insightful)
These companys will sell you a contract that DOES give your business recourse when something goes wrong.
Most businesses don't need enterprise software though, so they stick with linux, bsd, windows, and mac os.
It is still a bad problem, and there are alot of debian users, but just compared to all the linux distros out there, it is still only a percentage and under 100% (by Far under 100)
What you are saying is basically that because I personally can download an opensource program, change (aka break) it, and give it to someone, that opensource in whole is broken, untrustworthy, and bad.
Clearly that is just stupid.
I suppose if you don't trust openssl anymore, for a problem that effects only debian users, compared to MS's bugs that give admin access to anyone on the internet from win NT4 up to Vista SP1, then that is your call. (I am curious how you might justify that to your PHB though)
http://www.microsoft.com/technet/security/bulletin/MS08-001.mspx [microsoft.com]
Thats akin to saying "I no longer trust the guy at the store since he shorted me $0.03 in change, so i'm gunna just trust the crackwhore at the corner to by the same stuff from."
Re:The big question is.. (Score:3, Insightful)
The human kind of idiot (Score:3, Insightful)
A small coding mistake has had rather nasty and rather public consequences.
What does bear scrutiny is the process. In such a critical package should it really come down to just one guy? On such a critical package should Ubuntu not have their own man double checking everything rather than just accepting changes from Debian.
You can't make people infallible, but you improve the system to better mitigate such fallibility.
Re:Degree of Compromise? (Score:1, Insightful)
It's a 5-microsecond table lookup.