Debian Bug Leaves Private SSL/SSH Keys Guessable

SecurityBob writes "Debian package maintainers tend to very often modify the source code of the package they are maintaining so that it better fits into the distribution itself. However, most of the time, their changes are not sent back to upstream for validation, which might cause some tension between upstream developers and Debian packagers. Today, a critical security advisory has been released: a Debian packager modified the source code of OpenSSL back in 2006 so as to remove the seeding of OpenSSL random number generator, which in turns makes cryptographic key material generated on a Debian system guessable. The solution? Upgrade OpenSSL and re-generate all your SSH and SSL keys. This problem not only affects Debian, but also all its derivatives, such as Ubuntu." Reader RichiH also points to Debian's announcement and Ubuntu's announcement.

stupid stupid stupid

[spikedvodka (188722)]

Who did this? You don't remove the seeding... stupid

did I mention stupid?

this is how some of the old video games were "broken" despite using "random" numbers, the seed was always the same... leading to the same sequence of events

Re:stupid stupid stupid

[Omnifarious (11933)]

Who did this? You don't remove the seeding... stupid

That's what I was thinking too. What kind of idiot makes that sort of change?

Re:stupid stupid stupid

[penfold69 (471774)]

That's funny, I use the exact same seed on my luggage.

Re:stupid stupid stupid

[mikael (484)]

There was a lawsuit story some time ago about a Las Vegas casino vs. a regular punter, over the use of a blackjack game. The punter had found a cabinet unit with a frazzled RNG, so it kept dealing the same sequence of cards every time. The punter managed to memorize the sequence, thus guaranteeing a win every time. Of course, the casino wasn't too happy about this.

Re:stupid stupid stupid

[kensan (682362)]

The original change was tracked in the debian Bugtracking system at: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516 [debian.org]
The offending patch: http://svn.debian.org/viewsvn/pkg-openssl/openssl/trunk/rand/md_rand.c?rev=141&view=diff&r1=141&r2=140&p1=openssl/trunk/rand/md_rand.c&p2=/openssl/trunk/rand/md_rand.c [debian.org]
The fix: http://svn.debian.org/viewsvn/pkg-openssl/openssl/trunk/crypto/rand/md_rand.c?rev=300&view=diff&r1=300&r2=299&p1=openssl/trunk/crypto/rand/md_rand.c&p2=/openssl/trunk/crypto/rand/md_rand.c [debian.org]

Re:stupid stupid stupid

[SiliconEntity (448450)]

The patch that broke it was checked in by Kurt Roeckx [[e-mail]]. Don't know if he broke it or if he was just the gatekeeper for checkins. See:

http://svn.debian.org/viewsvn/pkg-openssl/openssl/trunk/rand/md_rand.c?rev=141&view=diff&r1=141&r2=140&p1=openssl/trunk/rand/md_rand.c&p2=/openssl/trunk/rand/md_rand.c [debian.org] which shows the change that introduced the bug; and its parent node:
http://svn.debian.org/viewsvn/pkg-openssl/openssl/trunk/rand/?rev=141#dirlist [debian.org] which shows the maintainer responsible.

From looking at this patch, I think this is what happened. valgrind complained about a rather unusual coding convention in ssleay_rand_bytes. This is a function that returns random data into a buffer. However, before writing into the buffer, it reads from the buffer and incorporates the old contents into the internal random state. valgrind complained about this use of an output buffer for input. Normally you would never want to use potentially uninitialized data like this, but in this case it is OK as all that is being done is the data is being folded into the random state. In the worst case, this can't hurt, and maybe it will help, if the old data had some randomness.

Anyway, valgrind complained about it, and the maintainer commented out the use of the buffer. That would actually be OK, it is not a big deal. But the implementor made a mistake, and also commented out another similar usage, in a different function, ssleay_rand_add. This was a huge mistake, as the purpose of ssleay_rand_add is to add randomness into the random state. In that function, buf is an INPUT buffer, and adding it into the random state is perfectly legitimate, in fact it is the whole purpose of the function. But apparently because it looked similar to the questionable usage in ssleay_rand_bytes, the maintainer commented out the code in ssleay_rand_add at the same time. (I don't know whether valgrind also complained about this second usage, but if so, it was mistaken. I think it's more likely that the maintainer just got fooled and over-generalized from the valgrind complaint.)

So the whole thing was an attempt to clean up code and remove warnings, but the fix accidentally broke a crucial piece of functionality, rendering ssleay_rand_add completely non-functional. So any attempt to add randomness into the RNG state, such as for seeding purposes, is ineffective. The random state ends up with much less variability, and therefore all the crypto is weak. As Bruce Schneier points out, bad crypto looks much the same as good crypto, so it took this long to notice it.

Hats off to the reviewer who picked up on the problem. Don't know who it was, but the same Kurt Roeckx [[e-mail]] checked in the fix.

Re:stupid stupid stupid

[Dekortage (697532)]

Exactly what I was thinking. But it could be interpreted multiple ways: (a) it was criminals; (b) it was terrorists; (c) it was Microsoft.

Re:stupid stupid stupid

[Yvanhoe (564877)]

How is that mutually exclusive ?

Re:What does "guessable" mean here?

[Yvanhoe (564877)]

From what I undestood, with my limited cryptographic knowledge and my total reluctance to dig into all the details of the sources, what happens is that when you asked your debian box to generate a key, it did so without integrating enough random data in its choice. That means that a brute force attack will not have to try every key to find yours but only a small subset, depending on the knowledge it has about your machine and its state when it generated the keys.
There is no such thing as a master key, but it would mean that several different servers may have generated the same "random" keys and that a clever attacker could create a dictionary of plausible keys (or, more plausibly, an algorithm to generate them) and try all of them on your server with a good chance to have one of them working. The dictionary would probably be huge however. A good measure would be to block an IP on SSH after a reasonnable number of failures (like 100)

Re:stupid stupid stupid

[bk2204 (310841)]

It's not seeding too often that's the problem, it's seeding with predictable data and expecting that data to be random. The time is very predictable, and contains very little entropy (randomness).

If you seed very often with data containing a lot of entropy (for example, radioactive decay), then there's no problem. It's also not a problem to add the time in the mix if you mark it as having almost no entropy.

Updated advisory from Ubuntu

[by Anonymous Coward]

Ubuntu got an updated advisory at http://www.ubuntu.com/usn/usn-612-2

Re:Updated advisory from Ubuntu

[by Anonymous Coward]

Linky link link: Updated Advisory with instructions [ubuntu.com]

The big question is..

[Idaho (12907)]

whether this can possibly be claimed to be an accident *dons tinfoil hat*.

But seriously, removing the code that seeds a random number generator? I can hardly imagine making such a change by accident. I may just lack a sufficiently colorful imagination, though.

(or, before resorting to conspiracy theories, we should probably ask ourselves first, "can this possibly be explained by simple stupidity?"

Re:The big question is..

[sciencewhiz (448595)]

It was reading from initialized memory to for the seed value, leading to valgrind warnings. See the original Debian bug report: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516 [debian.org]

Re:The big question is..

[Goaway (82658)]

Apparently, OpenSSL was using uninitialized memory as a source of randomness. This made valgrind complain about the program, and some enterprising programmer decided to fix it by clearing the memory before use.

Re:The big question is..

[IkeTo (27776)]

I think the problem is not the removal of the use of uninitialized memory as source of randomness. Instead it is the wrong way it is done, removing nearly all sources of randomness as a result. By the way, the fixed package still don't use the uninitialized memory as source of randomness.

OSS, only as good as the last developer?

[MosesJones (55544)]

First off I'm a big OSS supporter, yada, yada

But the point here is that the freedom that OSS gives you does require you to trust the whole distribution chain. In this case there was an added muppet who did something they shouldn't have thus rendering everything downstream insecure. OSS is great but it required great developers, given that it has take well over a year to get the advisory out it shows that the many eyes piece didn't work here, mainly because the eyes were looking at the original source not the botched packaging job.

The "easy to use" Linux box in the house uses Ubuntu and has this issue and like many people I didn't even think to check that the OpenSSL wasn't the REAL OpenSSL it was OpenSSL with muppet extensions. Maybe there needs to be some form of extension that warns that a package has been modified from its original source code and that the modification was done by "K. Frog" so you can determine whether to trust that package or look back to the source.

Or some sort of voting system on contributors (how very Web 2.0) so you can see how the people who touched your package were rated with the biggest weighting being given to the last person through the code (hand edited by Linus = 5 stars, hand edited by James Gosling = 5 stars, hand edited by the bloke who wrote clippy = 2 stars, hand edited by a bloke who removed a seed generator = 0 stars).

Having the code is great, but this makes me want to know much more about who last edited that code.

Re:OSS, only as good as the last developer?

[peragrin (659227)]

Ah so if the same thing happened from MSFT but no one noticed it does that mean closed source is better.

No software is perfect. when F/OSS screws up everything including the exact versions of the software where the bug began, until it is fixed is known. You know what/where/when/how, and most of the time why it happened.

With closed source software your considered lucky if you get a patch in a timely fashion.

Personally i would rather know what happened and when too.

What's the hurry?

[n0dna (939092)]

It was accidentally introduced in 2006... so that's what, another 14 years before it gets moved into 'stable'?

*grin*

How Frakin stupid can you be?

[nweaver (113078)]

"You fell for one of the classic blunders, the most famous being 'Never get involved in a land war in Asia' but only slightly less well known is 'Don't use poorly seeded pRNGs in cryptographic protocols!' HAHAHAHAHAHHAHAHAHAHHAHAHAHAHAHA!!!!

Re:How Frakin stupid can you be?

[EricR86 (1144023)]

BUTTERCUP: Who are you?

MAN IN BLACK: I am no one to be trifled with, that is all you ever need know.

BUTTERCUP: To think -- all that time it was your cryptographic protocol that was poorly seeded.

MAN IN BLACK: They were both poorly seeded. I spent the morning downloading a patch to build an immunity to keys being guessed.

2 years?

[by Anonymous Coward]

The seeding was removed and it wasn't noticed for TWO YEARS? In a distro as popular as Debian?

Re:2 years?

[jrothwell97 (968062)]

Hang on. There was a story here the other day about a bug in BSD that had been carried down to all its descendants (FreeBSD, OS X, NetBSD etc) for twenty-five years without anyone noticing. Two years is nothing.

comics

[by Anonymous Coward]

http://www.random.org/analysis/dilbert.jpg
http://www.xkcd.com/221/

Too early

[sakonofie (979872)]

I realized I probably should be legally required to have a morning cup of coffee before thinking because I am an idiot otherwise.

I wake up and what do I see first thing? That there is a problem with Debian's OpenSSH package and the /. article links to the following code snippet:

def init(pipeline, librarian):
          gst.debug_set_default_threshold(gst.LEVEL_ERROR)
- if gst.element_make_from_uri(gst.URI_SRC, "file://", ""):
+ if gst.element_make_from_uri(
+ gst.URI_SRC,
+ "file:///Sebastian/Droge/please/choke/on/a/bucket/of/cocks", ""):
                  global playlist
                  playlist = PlaylistPlayer(pipeline or "gconfaudiosink", librarian)
                  return playlist

Now I am thinking, "What exactly is going on here? Is choking on a bucket of cocks not a good source of randomness?"

A great filter

[Free the Cowards (1280296)]

Anyone who posts to this story saying that this is no big deal or telling other people to stop whining should simply be banned from Slashdot for life. If you cannot be bothered to read the article and you cannot be bothered to understand just what a serious vulnerability this is but you insist on insulting those who do, why should you be allowed to continue to post your ignorant bleating?

Re:A great filter

[Free the Cowards (1280296)]

Your SSL-enabled web sites spent two years being vulnerable to masquerade or man-in-the-middle attacks. Your SSH servers spent two years being vulnerable to same. Your SSH key pairs spent two years being easily crackable by anyone who happened to notice this vulnerability but didn't tell the world. I'm not entirely sure, but I think that SSL and SSH sessions either to or from the affected OSes may be crackable, and if so this would include traffic that was recorded at any time during the vulnerable period and can now be analyzed using knowledge of this bug to find out what data they carried.

This is a big deal. Maybe it doesn't affect you very much. It doesn't affect me at all, I've never run these distros. But you can bet right now that there are a lot of heavy Linux users out there going through a lot of trouble. This is going to be a bonanza for certificate authorities as everyone who generated SSL keys with these distros is going to need to purchase a new cert.

Surely this is not the only source of entropy!

[argent (18001)]

Going to http://www.links.org/?p=327 [links.org] I read...

OpenSSL happens to include a rare case when its OK, or even a good idea: its randomness pool. Adding uninitialised memory to it can do no harm and might do some good, which is why we do it.

Uninitialised data doesn't seem to be a good source of randomness to depend on, since depending on where it happens you may consistently end up with a buffer that previously contained all zeroes (or some default memory test pattern), the same part of the same shared library header, or a series of stack frames that for whatever reason happen to be the same frames on every run.

In fact I'd expect that separate runs of the same program with the same parameters and environment would leave the same junk on the stack every time.

So I would hope that they have some better source of entropy than unpredictable uninitialized data of dubious randomness.

So if this is really a serious problem, then it seems to me there's already a serious problem in OpenSSH.

Degree of Compromise?

[ewhac (5844)]

Does anyone have any back-of-the-envelope calculations as to how badly this compromises existing keys? That is to say, about how long is the brute-force lifetime shortened? If it's been shortened from the age of the known universe to 300 hours, then that's a problem I need to address fairly immediately. OTOH, if it's been shortened to one-quarter the age of the known universe, then I'm not going to deal with this before I've had more coffee...

Schwab

Re:It will be fixed

[gQuigs (913879)]

This problem isn't something you can just update your system to fix. This means the basis for all remote authentication on your Debian/Ubuntu machines is compromised until you go and fix it manually.

Re:It will be fixed

[imbaczek (690596)]

Not entirely true. Keys generated before the patch made it to the repos are safe - and I think quite a lot of debian boxes are old enough (I know I've got one.) There's a link in the advisory to a tool that checks if the keys are vulnerable.

This doesn't change the fact that this is a really serious fuckup. Debian lost quite some credibility in my eyes.

Re:It will be fixed

[Aaron Denney (123626)]

Actually, any DSA key *used* on a box with the bad SSL packages may be compromised:

From http://wiki.debian.org/SSLkeys

Additionally, some DSA keys may be compromised in the following situations:

        * key generated with broken openssl = bad
        * key generated with good openssl and used to ssh from a machine with bad ssl = bad
        * key generated with good openssl and used to ssh from a machine with good ssl = good

This is because the random numbers used during the signature process must also be good.

Re:It will be fixed

[truthsearch (249536)]

My wifes ssh key is "Debian compromised".

Talk about an easy key to guess! At least throw in some numbers or something. ;)

Re:It will be fixed

[dotancohen (1015143)]

My wifes ssh key is "Debian compromised"

Say thank you that your wife is at least that far. If I dare say ssh in the wife's presence, I get a vase thrown at me and she screams for me to shut the fuck up.

Re:It will be fixed

[Archangel Michael (180766)]

It shouldn't need fixing in the first place.

Debian people screwed up. This leaves a huge distaste in my mouth for Debian (and Ubuntu).

correct me if I'm wrong

[Deanalator (806515)]

From this log: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516 [debian.org]
It looks like openssl is pulling in "entropy" from uninitialized memory, causing valigrind to complain. The debian maintainer "fixed" this issue by memsetting the buffer to zero.

My question is, wouldn't I see the same behavior using grsecurity to scrub deallocated memory? From what I am seeing, this looks much more like the fault of the openssl team.

Re:It will be fixed

[eean (177028)]

Check out this bug report and tell me they weren't just being arrogant:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516 [debian.org]

Says the first comment to the patch, in regards to what is being reversed:
"What it's doing is adding uninitialised numbers to the pool to create random numbers."

OpenSSL having more-random numbers vs. Valgrind looking good. And Valgrind won?

The consensus in the bug report seems to be not to do it, but then someone adds the patch anyways.

Re:It will be fixed

[rhavenn (97211)]

I'm sure the problem will be fixed if the developers acknowledge that the problem exists. Not a big worry.

No, but it shouldn't have been changed in the first place. Debian needs to stick their ego up their ass sometimes and just let the people who wrote the software do the coding vs. sticking their own code in in places they don't fully understand. This and their attitude of licensing and not reporting changes back upstream is a stupidly annoying habit.

note: When I have to run Linux instead of a BSD it's Debian and/or Kubuntu all the way since the benefits outweigh the negatives, but it's still an annoying habit of theirs.

Re:It will be fixed

[Omnifarious (11933)]

I'm sure the problem will be fixed if the developers acknowledge that the problem exists. Not a big worry.

Yes, it is a big worry because any keys generated with this package are now potentially suspect. This means that anybody who's used Debian or a Debian derived distribution like Ubuntu needs to go back and destroy all host and personal keys generated since 2006. All of those keys are potentially guessable.

And that's a real vulnerability. Early versions of Netscape's SSL implementation (the first SSL implementation) were trivially crackable because of just such a vulnerability [berkeley.edu].

Don't underestimate corporate arse-covering

[zooblethorpe (686757)]

If some coder did this at a company at least I'm pretty confident they'd get their ass fired, but with open source it's basically "whoops, my bad."

I, on the other hand, strongly suspect that any similar mistake at a major software corporation would in all probability be quietly ignored, if it were even noticed at all -- and if it were instead deemed enough of a public relations risk to warrant dealing with, the company would likely just silently push an update to correct the problem for future users, leaving anyone using extant keys with their arses hanging in the breeze.

But maybe I'm just being overly cynical. :-\

Cheers,

Re:It will be fixed

[2short (466733)]

Basic cryptographic services have been compromised for a year and your analysis is to assume on faith that it's open source so it will be fixed, so no problem?

If someone stole your crypto keys and has had them for a year...

How thoroughly might they have compromised your system by now?
How many passwords might they have stolen that you use on other systems?
What else might they have done that will give them access in the future even after you fix this?

Just regenerate your keys and no problem? The problem that guessable keys are generated will undoubtably be fixed asap, if not already. The problem that this has been the case for the last year will not be, and is a big worry.

Re:It will be fixed

[Jellybob (597204)]

Downloading the patch is step one - you also need to regenerate any certificates made with OpenSSL since 2005, since they can't be guaranteed to be secure.

This has the potential to turn into a huge pain in the arse for Debian based shops, who will need to reissue SSL certificates, SSH keys, and a whole host of other essential elements of their security infrastructure.

Re:It will be fixed

[MSG (12810)]

I think the point was that if the Debian maintainer had submitted his patch upstream, he'd have been told exactly how stupid it was.

Re:It will be fixed

[MikeyO (99577)]

I think the point was that if the Debian maintainer had submitted his patch upstream, he'd have been told exactly how stupid it was.

Actually, here [marc.info] is where the Debian maintainer brought this up with upstream, and he was not told how stupid it was, in fact the Debian maintainer says, "the pool might receive less entropy." and upstream says, "If it helps with debugging, I'm in favor of removing them."

Re:You stupid god damned open sourcers

[clang_jangle (975789)]

Windows here I come. At least someone would be accountable for shit work like this.

Yeah, like all those times when MS cut checks for all their customers whose computers were compromised! Oh, wait...

Re:Of course...

[Oxy the moron (770724)]

Quit being a cry baby and run 'apt-get upgrade' already. It would have taken you less time than to come in here complain.

... and regenerate all the keys, yes? It isn't quite as simple as you suggest...

"All OpenSSH and X.509 keys generated on such systems must be considered untrustworthy, regardless of the system on which they are used, even after the update has been applied."

Re:Of course...

[goofy183 (451746)]

But that doesn't fix all of the machines that my public key (generated on Ubuntu) resides on that do not run Ubuntu. So yes fixing Ubuntu machines is easy, it is a PITA to have to go and re-upload my public key to all of these hosts.

Re:Of course...

[Just Some Guy (3352)]

So yes fixing Ubuntu machines is easy, it is a PITA to have to go and re-upload my public key to all of these hosts.

Especially when they won't work anymore [ubuntu.com]:

Once the update is applied, weak user keys will be automatically rejected where possible (though they cannot be detected in all cases). If you are using such keys for user authentication, they will immediately stop working and will need to be replaced (see step 3).

The GP poster said the same thing, but this is for the benefit of other readers who were skeptical that any such policy was in place.

So, basically, once you upgrade, you'll have no apparent way to access your other machines [1] to upload your new key. That's just spiffy!

[1] Uninstall openssh-blacklist first.

Re:To non-IT people

[Sun.Jedi (1280674)]

This is on the SSH server side. If you are a casual desktop user, you don't have to do anything.

Yes, a good observation for IM/Email/Youtube/Facebook crowd... but how many others ssh into their home machine? I'd wager the ability to ssh into a home box is one of the better perks to running linux@home.

Re:i wondered what was going on

[sumdumass (711423)]

No, the random text your put in can be random or not. What they were talking about is a number generator that was used or applied to an algorithm that decided the prime of your key in so that the same algorithm wouldn't be used on two machines on the first then second keys. Without the random number generator, you could theoretically guess the algorithm used to generate the key based of the actions of another install and then decipher the key to gain access to whatever it was protecting.

To simplify and generalize it, if every machine uses X+1*256 to get a 256 bit key equal to 768, then you could reverse that and know X would =2 (3*256=768) and fake the credentials. The random number generator should change that to X+R*256 which make reversing the key harder because you can only solve to X+R=3 now. In practice, it will be a really larger number and a lot different process though. I can't say that I fully understand it but that simplification should show the difference well enough to give an Idea of where the problem is.