Debian Bug Leaves Private SSL/SSH Keys Guessable

SecurityBob writes "Debian package maintainers tend to very often modify the source code of the package they are maintaining so that it better fits into the distribution itself. However, most of the time, their changes are not sent back to upstream for validation, which might cause some tension between upstream developers and Debian packagers. Today, a critical security advisory has been released: a Debian packager modified the source code of OpenSSL back in 2006 so as to remove the seeding of OpenSSL random number generator, which in turns makes cryptographic key material generated on a Debian system guessable. The solution? Upgrade OpenSSL and re-generate all your SSH and SSL keys. This problem not only affects Debian, but also all its derivatives, such as Ubuntu." Reader RichiH also points to Debian's announcement and Ubuntu's announcement.

Updated advisory from Ubuntu

[Anonymous Coward]

Ubuntu got an updated advisory at http://www.ubuntu.com/usn/usn-612-2

Re:It will be fixed

[Anonymous Coward]

Yes it's already fixed. I was in the middle downloading the update when when I saw this article.

Re:It will be fixed

[Omnifarious]

I'm sure the problem will be fixed if the developers acknowledge that the problem exists. Not a big worry.

Yes, it is a big worry because any keys generated with this package are now potentially suspect. This means that anybody who's used Debian or a Debian derived distribution like Ubuntu needs to go back and destroy all host and personal keys generated since 2006. All of those keys are potentially guessable.

And that's a real vulnerability. Early versions of Netscape's SSL implementation (the first SSL implementation) were trivially crackable because of just such a vulnerability [berkeley.edu].

Re:It will be fixed

[Anonymous Coward]

Um.... it doesn't matter if it's been patched, you still have to regenerate all your keys.

There most certainly is something to see here, this is kind of a big deal.

Re:i wondered what was going on

[Anonymous Coward]

This has nothing to do with the vulnerability being discussed here.

Re:The big question is..

[sciencewhiz]

It was reading from initialized memory to for the seed value, leading to valgrind warnings. See the original Debian bug report: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516 [debian.org]

Re:stupid stupid stupid

[John Meacham]

doh! posted the wrong URL

http://securepoint.com/lists/html/bugtraq/2007-02/msg00056.html [securepoint.com]

To non-IT people

[Thelasko]

This is on the SSH server side. If you are a casual desktop user, you don't have to do anything.

Re:It will be fixed

[Jellybob]

Downloading the patch is step one - you also need to regenerate any certificates made with OpenSSL since 2005, since they can't be guaranteed to be secure.

This has the potential to turn into a huge pain in the arse for Debian based shops, who will need to reissue SSL certificates, SSH keys, and a whole host of other essential elements of their security infrastructure.

Re:It will be fixed

[Omnifarious]

i was sure that netscape problem was due to the US crypto export regs (the silly encryption=munitions thing) that limited the encryption to 40-bit keys.

Then you are mistaken. Read the linked to paper. N bit security doesn't protect you much when you can guess N-10 bits of the N bits.

Re:To non-IT people

[maxume]

It affects all https connections, even client https connections. They are probably transient enough to mitigate the issue, but they are impacted.

Re:Updated advisory from Ubuntu

[Anonymous Coward]

Linky link link: Updated Advisory with instructions [ubuntu.com]

Re:The big question is..

[gowen]

And the ridiculous thing is, in that thread you quote, at least one person is telling the maintainer that they've done a seriously wrong thing, but the Debian OpenSSL maintainer would rather mindlessly eliminate the valgrind warnings than actually think about the reasons for those warnings.

(Oh, and you meant "UNinitialized" memory.)

Re:To non-IT people

[jdowland]

Uhhm, no. If you generated a key using the affected packages (ssh-keygen), as a user, your key needs to be replaced.

Re:The big question is..

[Goaway]

Apparently, OpenSSL was using uninitialized memory as a source of randomness. This made valgrind complain about the program, and some enterprising programmer decided to fix it by clearing the memory before use.

Re:i wondered what was going on

[sumdumass]

No, the random text your put in can be random or not. What they were talking about is a number generator that was used or applied to an algorithm that decided the prime of your key in so that the same algorithm wouldn't be used on two machines on the first then second keys. Without the random number generator, you could theoretically guess the algorithm used to generate the key based of the actions of another install and then decipher the key to gain access to whatever it was protecting.

To simplify and generalize it, if every machine uses X+1*256 to get a 256 bit key equal to 768, then you could reverse that and know X would =2 (3*256=768) and fake the credentials. The random number generator should change that to X+R*256 which make reversing the key harder because you can only solve to X+R=3 now. In practice, it will be a really larger number and a lot different process though. I can't say that I fully understand it but that simplification should show the difference well enough to give an Idea of where the problem is.

Re:It will be fixed

[eean]

Check out this bug report and tell me they weren't just being arrogant:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516 [debian.org]

Says the first comment to the patch, in regards to what is being reversed:
"What it's doing is adding uninitialised numbers to the pool to create random numbers."

OpenSSL having more-random numbers vs. Valgrind looking good. And Valgrind won?

The consensus in the bug report seems to be not to do it, but then someone adds the patch anyways.

Re:It will be fixed

[imbaczek]

Not entirely true. Keys generated before the patch made it to the repos are safe - and I think quite a lot of debian boxes are old enough (I know I've got one.) There's a link in the advisory to a tool that checks if the keys are vulnerable.

This doesn't change the fact that this is a really serious fuckup. Debian lost quite some credibility in my eyes.

Re:stupid stupid stupid

[kensan]

The original change was tracked in the debian Bugtracking system at: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516 [debian.org]
The offending patch: http://svn.debian.org/viewsvn/pkg-openssl/openssl/trunk/rand/md_rand.c?rev=141&view=diff&r1=141&r2=140&p1=openssl/trunk/rand/md_rand.c&p2=/openssl/trunk/rand/md_rand.c [debian.org]
The fix: http://svn.debian.org/viewsvn/pkg-openssl/openssl/trunk/crypto/rand/md_rand.c?rev=300&view=diff&r1=300&r2=299&p1=openssl/trunk/crypto/rand/md_rand.c&p2=/openssl/trunk/crypto/rand/md_rand.c [debian.org]

Re:stupid stupid stupid

[SiliconEntity]

The patch that broke it was checked in by Kurt Roeckx [[e-mail]]. Don't know if he broke it or if he was just the gatekeeper for checkins. See:

http://svn.debian.org/viewsvn/pkg-openssl/openssl/trunk/rand/md_rand.c?rev=141&view=diff&r1=141&r2=140&p1=openssl/trunk/rand/md_rand.c&p2=/openssl/trunk/rand/md_rand.c [debian.org] which shows the change that introduced the bug; and its parent node:
http://svn.debian.org/viewsvn/pkg-openssl/openssl/trunk/rand/?rev=141#dirlist [debian.org] which shows the maintainer responsible.

From looking at this patch, I think this is what happened. valgrind complained about a rather unusual coding convention in ssleay_rand_bytes. This is a function that returns random data into a buffer. However, before writing into the buffer, it reads from the buffer and incorporates the old contents into the internal random state. valgrind complained about this use of an output buffer for input. Normally you would never want to use potentially uninitialized data like this, but in this case it is OK as all that is being done is the data is being folded into the random state. In the worst case, this can't hurt, and maybe it will help, if the old data had some randomness.

Anyway, valgrind complained about it, and the maintainer commented out the use of the buffer. That would actually be OK, it is not a big deal. But the implementor made a mistake, and also commented out another similar usage, in a different function, ssleay_rand_add. This was a huge mistake, as the purpose of ssleay_rand_add is to add randomness into the random state. In that function, buf is an INPUT buffer, and adding it into the random state is perfectly legitimate, in fact it is the whole purpose of the function. But apparently because it looked similar to the questionable usage in ssleay_rand_bytes, the maintainer commented out the code in ssleay_rand_add at the same time. (I don't know whether valgrind also complained about this second usage, but if so, it was mistaken. I think it's more likely that the maintainer just got fooled and over-generalized from the valgrind complaint.)

So the whole thing was an attempt to clean up code and remove warnings, but the fix accidentally broke a crucial piece of functionality, rendering ssleay_rand_add completely non-functional. So any attempt to add randomness into the RNG state, such as for seeding purposes, is ineffective. The random state ends up with much less variability, and therefore all the crypto is weak. As Bruce Schneier points out, bad crypto looks much the same as good crypto, so it took this long to notice it.

Hats off to the reviewer who picked up on the problem. Don't know who it was, but the same Kurt Roeckx [[e-mail]] checked in the fix.

Re:The big question is..

[IkeTo]

I think the problem is not the removal of the use of uninitialized memory as source of randomness. Instead it is the wrong way it is done, removing nearly all sources of randomness as a result. By the way, the fixed package still don't use the uninitialized memory as source of randomness.

Re:Why open source doesn't work for business

[ledow]

Hello troll,

You have no recourse unless you have PAID to have one, like with almost everything else in the world. The GPL specifically states that it's not a warranty or guarantee that things will work.

However, if you were using it in a business, I would hope that you either a) hired your own programmers to work with the code or b) bought a support package/liability clause from someone like Red Hat. In which case it would be down to your programmers or Red Hat respectively. But we're not talking about Red Hat. Or any of the other big-name, support-contract-and-some-sort-of-indemnity-clause-included distros. They don't have this problem, presumably because they are not that stupid.

We're talking about Debian. Got a support contract / liability agreement with Debian? No? Bad luck. It's a bit like asking Microsoft to accept responsibility for your pirate copy of Windows, then, isn't it?

Re:stupid stupid stupid

[bk2204]

It's not seeding too often that's the problem, it's seeding with predictable data and expecting that data to be random. The time is very predictable, and contains very little entropy (randomness).

If you seed very often with data containing a lot of entropy (for example, radioactive decay), then there's no problem. It's also not a problem to add the time in the mix if you mark it as having almost no entropy.

Re:Of course...

[travalas]

On ubuntu.... according to the vulnerability description.. Once the update is applied, weak user keys will be automatically rejected where possible (though they cannot be detected in all cases). If you are using such keys for user authentication, they will immediately stop working and will need to be replaced (see step 3). OpenSSH host keys can be automatically regenerated when the OpenSSH security update is applied. The update will prompt for confirmation before taking this step.

Re:Of course...

[goofy183]

But that doesn't fix all of the machines that my public key (generated on Ubuntu) resides on that do not run Ubuntu. So yes fixing Ubuntu machines is easy, it is a PITA to have to go and re-upload my public key to all of these hosts.

Re:It will be fixed

[MikeyO]

I think the point was that if the Debian maintainer had submitted his patch upstream, he'd have been told exactly how stupid it was.

Actually, here [marc.info] is where the Debian maintainer brought this up with upstream, and he was not told how stupid it was, in fact the Debian maintainer says, "the pool might receive less entropy." and upstream says, "If it helps with debugging, I'm in favor of removing them."

Re:It will be fixed

[Aaron Denney]

Actually, any DSA key *used* on a box with the bad SSL packages may be compromised:

From http://wiki.debian.org/SSLkeys

Additionally, some DSA keys may be compromised in the following situations:

        * key generated with broken openssl = bad
        * key generated with good openssl and used to ssh from a machine with bad ssl = bad
        * key generated with good openssl and used to ssh from a machine with good ssl = good

This is because the random numbers used during the signature process must also be good.

Re:Surely this is not the only source of entropy!

[Anonymous Coward]

The uninitialised memory there isn't the only source they use, they just use it if it is there because it doesn't matter, and at worst doesn't make the data less random. They then call a randomisation routine. The broken patch commented out both calls. The fix doesn't restore the proper state as it doesn't fully undo the change that broke it, so in due course there will be another fix later this week with a full fix.

Don't mess with code you don't understand, especially when it is so important.

If you used this code to generate certificate requests, do you need to repurchase your SSL certificates?

Re:What does "guessable" mean here?

[Yvanhoe]

From what I undestood, with my limited cryptographic knowledge and my total reluctance to dig into all the details of the sources, what happens is that when you asked your debian box to generate a key, it did so without integrating enough random data in its choice. That means that a brute force attack will not have to try every key to find yours but only a small subset, depending on the knowledge it has about your machine and its state when it generated the keys.
There is no such thing as a master key, but it would mean that several different servers may have generated the same "random" keys and that a clever attacker could create a dictionary of plausible keys (or, more plausibly, an algorithm to generate them) and try all of them on your server with a good chance to have one of them working. The dictionary would probably be huge however. A good measure would be to block an IP on SSH after a reasonnable number of failures (like 100)

Re:It will be fixed

[dondelelcaro]

The consensus in the bug report seems to be not to do it, but then someone adds the patch anyways.

The package maintainer first checked with upstream [marc.info] about the best way to resolve this. In retrospect, it's clear that upstream didn't either understand what was being asked, or what the code was doing. In any event, another Debian Developer, Luciano Bello, later found the problem and resolved it.

Re:Of course...

[Just Some Guy]

So yes fixing Ubuntu machines is easy, it is a PITA to have to go and re-upload my public key to all of these hosts.

Especially when they won't work anymore [ubuntu.com]:

Once the update is applied, weak user keys will be automatically rejected where possible (though they cannot be detected in all cases). If you are using such keys for user authentication, they will immediately stop working and will need to be replaced (see step 3).

The GP poster said the same thing, but this is for the benefit of other readers who were skeptical that any such policy was in place.

So, basically, once you upgrade, you'll have no apparent way to access your other machines [1] to upload your new key. That's just spiffy!

[1] Uninstall openssh-blacklist first.

Passwords may be compromised, too

[Krischi]

It is even worse than this. Passwords that were used on a server with a weak DSA key may be compromised, as well:

From http://wiki.debian.org/SSLkeys [debian.org]

Re:stupid stupid stupid

[ivanjager]

Wow, that was still a really stupid patch. There was an #ifndef PURIFY there for a reason. It's because the openssl authors knew that line would cause trouble in a memory debuger like Purify or Valgrind. http://en.wikipedia.org/wiki/IBM_Rational_Purify [wikipedia.org]

Seriously guys, take a few minutes to understand the code before hacking on it. If it was written by a n00b, you need to understand it to make sure they didn't do other stuff wrong. If it was written by someone competent (which is probably the case for openssl), then you need to make sure you're not the one making the mistake. In this case, to make it work with valgrind, the patch should have been to debian/rules, simply changing the way Configure was called.

Re:What does "guessable" mean here?

[LordSnooty]

SSHDFilter [liv.ac.uk] may help in this regard. Everyone should be taking steps to limit brute-force ssh attacks even without this current issue.

Re:It will be fixed

[eean]

Ben Laurie of OpenSSL/Apache puts it into some context:
http://www.links.org/?p=327#comment-176642 [links.org]

Obviously some of the OpenSSL devs probably should've been like "zOMG, SITUATION FUBAR", but it wasn't a formal code review being requested, more of a "hey, what do you think of this and this?" and the patch was never submitted to upstream.

Some important information

[xtronics]

You will want to download this tool to check each system:

http://security.debian.org/project/extra/dowkd/dowkd.pl.gz [debian.org]

To use it:

$ perl dowkd.pl user

$ perl dowkd.pl host localhost

See http://lists.debian.org/debian-security-announce/2008/msg00152.html [debian.org]

Why isn't such a scan isn't part of the package update? http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=481106 [debian.org]

262148 possible keypairs?

[multi io]

reading the weak key detector script...

...it looks like the affected Debian OpenSSHs can generate only 262148 (or something) distinct keypairs. Can this be true? If it is, this is fucking serious.

Re:Degree of Compromise?

[multi io]

If I read the published "weak key detector" script correctly, Debian OpenSSHs will always generate one out of a fixed set of 262148 possible keypairs. Do the math yourself. If you know those keys, this is a 5-minute brute force attack.

Re:stupid stupid stupid

[Anonymous Coward]

Looks like Kurt didn't feel quite sure about the patch from the start:

http://marc.info/?l=openssl-dev&m=114651085826293&w=2 [marc.info]

In a follow-up message, Ulf Möller of OpenSSL approved of the patch. He at least should have known better.

Re:It will be fixed

[KutuluWare]

I think you misread his statement. He didn't say your "machine" was compromised. He said that the underlying authentication mechanism was compromised, which is quite different. If I were to leave my car doors unlocked, the engine running, and the alarm disabled, then the security system on my car would be compromised; but until someone came along to actually hop in and drive away with it, the car itself would still be working properly.

Similarly, the fact that there is a problem with the OpenSSH package means that the security mechanism that is guarding access to your system is no longer as secure as it was previously thought -- that makes it compromised. The individual systems may or may not have been compromised themselves by way of the compromised security mechanism, but continuing to use OpenSSL knowing it (the key gen algorithm, specifically) is less secure than it should be is just asking for trouble.

Re:correct me if I'm wrong

[annodomini]

No, I don't think you're quite correct. The original patch in that thread is not what actually was applied. There are two functions, one for seeding the pseudo-random number generator, and one for getting random data out of it. The one for getting random data out was actually mixing in some uninitialized data into the entropy pool. This is not particularly harmful, but not particularly helpful either, and it caused Valgrind to complain. So, the maintainer removed that line, but also removed the line that mixes in the value provided to seed the random number in the first place (maybe it was being called with unitialized data at some point, too).

See my comment here for more details: http://reddit.com/info/6j7a9/comments/c03zxko [reddit.com]

Re:It will be fixed

[Anonymous Coward]

It shouldn't need fixing in the first place.

Debian people screwed up. This leaves a huge distaste in my mouth for Debian (and Ubuntu).

Volunteers not maintaining the free product that you've decided to trust your security with properly? Get over it.

Re:Kurt Roeckx (debian) mail to openssl-dev list

[grumbel]

i thought that Debian had some kind of policies about testing each package before committing changes in testing/stable branches.

This is a common myth, but Debian doesn't do any explicit testing, they pack things into unstable and then just let them go into testing/stable if no bug reports come in. There is no standard code audit or anything going on.

Re:It will be fixed

[Sir_Real]

No, but it shouldn't have been changed in the first place. Debian needs to stick their ego up their ass sometimes and just let the people who wrote the software do the coding vs. sticking their own code in in places they don't fully understand.

WTF? This bug is the result of a Debian guy saying "Hey, this doesn't look right, can you fix it?" The PACKAGE MAINTAINER (Not the DEBIAN GUY) "fixed" it for Debian (though for obvious reasons that fix won't propogate any further) and got us here. Speaking of "don't fully understand..." Would you maybe read .... prior to ....

ah fuck it
this is slashdot

Re:It will be fixed

[noahm]

No, he posted a question openssl-dev, which is a mailing list for people writing software that uses OpenSSL

Incorrect, no matter what Ben claims. According to http://www.openssl.org/support/ [openssl.org], openssl-dev is for the developers of openssl itself. To quote the list description: Discussions on development of the OpenSSL library. Not for application development questions!

So yes, perhaps Kurt could have been more explicit in his description of what he was trying to do, but he was definitely using the appropriate address to reach the developers.

noah

Re:The big question is..

[Goaway]

No, *this* bug. Nothing should ever depend on the value of uninitialized memory, and if clearing it produced a security flaw then there was one there already. Unless it was actually cleared after being filled from /dev/random or something, but then there shouldn't have been a valgrind warning in the first place.

Uninitialized memory was only being used as an extra source of randomness. There was a compile-time flag to turn this off to make debuggers stop complaining.

However, Debian didn't use this flag, but instead tried to comment out the code themselves. And they commented out too much, breaking the whole thing completely.

Re:Passwords may be compromised, too

[embsysdev]

Thanks, that was very informative. Also, here is the text from a Gutsy upgrade:

Vulnerable host keys will be regenerated

    Some of the OpenSSH server host keys on this system were generated with a version of OpenSSL that had a broken random number generator. As a result, these host keys are from a well-known set, are subject to brute-force attacks, and must be regenerated.

    Users of this system should be informed of this change, as they will be prompted about the host key change the next time they log in. Use 'ssh-keygen -l -f HOST_KEY_FILE' after the upgrade has changed to print the fingerprints of the new host keys.
    The affected host keys are: /etc/ssh/ssh_host_rsa_key /etc/ssh/ssh_host_dsa_key

    User keys may also be affected by this problem. The 'ssh-vulnkey' command may be used as a partial test for this. See /usr/share/doc/openssh-server/README.compromised-keys.gz for more details.

Re:Does SSH not use /dev/random?

[vrmlguy]

The problem is, OpenSSL needed a buffer in which to XOR several sources of randomness, including /dev/random. OpenSSL didn't bother initializing the buffer, because doing so just eats CPU cycles, and a possibly random buffer is marginally better than a non-random one. However, the Debian patch removed *all* of the randomness except for the PID, leaving the user with one of just 262148 (or so) distinct keypairs. The tool that checks if your keys have been conpromised has a list of all of those keypairs and checks if your keypair appears.

To update the host keys on servers

[vrillusions]

Since they didn't mention how to do this in the security advisory, or give you a warning message when you apply the update that you should regen your keys here's how to regenerate the host keys. Either log in as root or put sudo before each of these (this should work remotely, I didn't have problems but YMMV)

rm -f /etc/ssh/ssh_host_dsa_key
rm -f /etc/ssh/ssh_host_rsa_key
ssh-keygen -b 1024 -N '' -f /etc/ssh/ssh_host_dsa_key -t dsa
ssh-keygen -b 2048 -N '' -f /etc/ssh/ssh_host_rsa_key -t rsa
/etc/init.d/ssh restart

You'll then be asked about the key changing next time you log in.

I just spent all day updating servers. Doesn't sound so bad till you realize all these servers also had client keys used for remote backups that had to be recreated and verified, etc.

Re:262148 possible keypairs?

[multi io]

From the same script:

Note that the blacklist by dowkd may be incomplete; it is only intended as a quick check.

With a statement as vague as that, there could be several orders of magnitude more potential keys.

I have an affected system here, and when I use its ssh-keygen to generate a few hundred keys, the script reports them all as "weak"[1], i.e. they are all in that set of 262,000. Looks like the script is at least quite good at what it does, and/or if there are more potential keys, they are rarely generated? When I do the same on a Fedora box here, none of them are reported, as expected.

260,000 would be bad though. At one test per second, that's just over 3 days. Of course, if you blacklist hosts after a certain number of failed authorization attempts, you can greatly increase the difficulty of exhausting the keyspace for a potential cracker.

...unless the attacker can obtain the public key directly, which he should be able to do frequently (it's called "public key" for a reason). For example, you can query a host for its public host (SSH) key without a problem. So, unless I'm missing something, you could just obtain that key and match it locally against those 262,000 keys (takes a minute at most, if you have all those keys) in order to find out the private key.

[1]
$ for i in `seq 1 500`; do ssh-keygen -N '' -t rsa -f key$i; done
$ ./dowkd.pl file key*.pub | grep 'weak key' | wc -l
500
$

Re:Updated advisory from Ubuntu

[this great guy]

Actually Ubuntu released 3 advisories for this vuln, because the vulnerable code is included in 3 packages. Their permanent links are: OpenSSL [ubuntu.com], OpenSSH [ubuntu.com] and OpenVPN [ubuntu.com].

Re:It will be fixed

[SiliconEntity]

According to http://www.openssl.org/support/ [openssl.org], openssl-dev is for the developers of openssl itself. To quote the list description: Discussions on development of the OpenSSL library. Not for application development questions!

Right, and further, one of the participants in the discussion was Ulf Moeller, who is one of the main crypto developers in OpenSSL. Unfortunately nobody apparently bothered to check whether the two lines proposed for removal were both involved in this questionable behavior of folding uninitialized data into the random state. Actually, only one of them was being used that way, the other was adding very important data to the random state. The OpenSSL people never caught the fact that the proposed change to the two lines was going to kill the security of the system, even though they had all the information available to do so.

Re:The issue is a bit more complicated...

[Free the Cowards]

This has been explained elsethread, but just in case anyone wanders into this one....

The problem is not the fact that the uninitialized memory seeding was removed. The problem is that the removal was done in an incompetent and destructive manner. Rather than remove only the uninitialized memory seeding, the maintainer managed to remove all seeding.

Reading uninitialized memory as part of random seeding isn't very useful but it's not bad either, since at worst the data is predictable, and OpenSSL mixes it with lots of other randomness. But when you botch your change and completely destroy any randomness present in the system, this is not the fault of the software you were trying to "fix".

Re:Once again, shoddy code leads to errors

[Anonymous Coward]

You mean besides the comment explaining how the code would break in a similar code checker, and the configuration #define offered to allow the code to be removed so that you could use code checkers on it?

Yes, truly horrible.

The code explained what was going on perfectly. The Debian maintainers were just not paying attention.

Don't forget the authorized_keys files

[halsathome]

Regenerating keys is fine, but the important step is clearing out the authorized_keys file that contains the old keys :-). At the same time take the opportunity to clear out the keys belonging to that long-gone co-worker.

If you try logging in with a new key where an old one is expected, you will get an error and remove the old one from authorized_keys. When the client is never in use however, the hole might stay in there for a long time ...

Ubuntu Dapper 6.06 LTS fine, too

[zzatz]

As with Debian Sarge, Dapper uses a version that predates the dangerous patch.

Re:Some important information

[xtronics]

To change-out server keys:

Had to change out all the other machines.

if you

$ cd /etc/ssh
$ rm ssh_host*
$ wajig reconfigure openssh-server

changing user keys - become user and ssh-keygen

You may have to fiddle with knownhosts and authorized_hosts as documented many places..

Re:What does "guessable" mean here?

[elronxenu]

What you are doing here is dropping only one packet per minute (with a maximum burst of 5 by default). The opposite of what you intended.

To make it work the way you intended, you need two rules:

iptables -I INPUT 1 -p tcp --dport 22 --syn -m limit --limit 1/minute -j ACCEPT
iptables -I INPUT 2 -p tcp --dport 22 --syn -j DROP

The limit rule runs first, accepting an average of 1 syn packet per minute. If the rate is exceeded then the rule fails and the DROP rule runs, which drops all further attempts to connect to ssh.

Re:It will be fixed

[AigariusDebian]

I cann't claim to 100% understand the situation but after glancing trough the logs of the discussions and of the patches the conclusion I came to was this - OpenSSL used supposed randomness of the uninitialized memory as an added source of entropy (interesting hack, but not an example of good coding as such). Valgring caught that problem and the Debian maintainer during a cleanup fixed it. Making such a fix can be considered a preventive step against possible attack vectors by poisoning the uninitialized memory. He took it up to upstream, they did not raise red flags, but did not quite merge the âclean upâ(TM) patch either. It fell through the cracks.

The problem is that in the same file, in another function all other sources of entropy were being merged into the pool of randomness using exactly the same code line as the one code line flagged by Valgrind. The maintainer assumed that the second code line has a similar function to the first and commented that one as well. AFAIK that also did not show up in the emails to the upstream list.

So we have:

        * Upstream using clever hacks that rely on uninitialized memory having some randomness to it
        * Upstream using same code and same variable names to describe different things
        * Upstream having no comments in the code explaining the two things above
        * Maintainer slightly over-generalizing a change
        * A bug slipping trough the cracks in the review processes
        * Another Debian Developer discovering the bug and recognizing its significance despite all of the above
        * Debian project coming out and admitting all of the above and scrambling to get fixes out to its users ASAP

I am impressed by the swift action of the people involved in fixing this. And while I think everyone can find some lesson be learned here, I think this is another good example of free software in action. And I hope that in the aftermath of this we will find ways to prevent this from happening in the future without stifling our progress.

http://www.aigarius.com/blog/2008/05/14/too-similar-to-be-different/ [aigarius.com]

Re:It will be fixed

[Archangel Michael]

Why stop with just compiling source?

Why not read every line, of every piece of code you compile, to make sure it wasn't tampered with?

Why not write your own code, code you review yourself and compile and maintain for your own use?

Unless you write, review, compile every piece of software source yourself, you have to trust someone to do at least one of them. At that point, you trust someone to do it for you, and you're already where your compaint lies. Its just that you don't realize it yet.

Compiling code you've never looked at is just as risky as trusting that someone to write the code for you. In fact, I suspect that the Debian Source would have compiled with the same problem, had you compiled it yourself.

So no, compiling from source (Debian) wouldn't have solved this problem, unless you could realize and fix the changes from the Debian Source and the Master Source yourself. But you didn't, someone else did.

Your point is kind of moot, isn't it?

MOD PARENT DOWN

[jareds]

This is dangerously misleading! Blocking IPs after a reasonable number of failures stops only one particular attack against SSH. It prevents someone from logging into a user account that has a compromised-Debian-generated SSH public key by brute force, trying all possible compromised-Debian key pairs. There are a number of other attacks.

The most serious involve the SSH host key. The public host key is given to anyone who attempts to connect to the machine, whether they succuessfully authenticate or not. This must be the case -- giving out the public key is part of establishing an encrypted connection, and you must have encryption before you send your password or something. An attacker can connect ONCE to your server and, from the public host key, lookup the corresponding private key offline in microseconds. They can then decrypt any SSH communications they can sniff where that server is the server. You have little more security than telnet. They can perform a man-in-the-middle attack on the server. Even if a user is carefully using public-key authentication with something other than a compromised-Debian-generated key rather than password authentication, if they connect to a server with a compromised-Debian-generated host key, they might be connecting to a MITM attacker who can then perform any action on the server as said user.

Public-key authentication is still seriously vulnerable even if you limit the number of attempts. Normally, I can generate a key pair on machine A and set up machines B and C to allow me to log in with that public key, and a root attacker on machine B or C will not be able to access the other machine with my identity. If machine A is a compromised-Debian machine, this no longer holds. Anyone with access to the public key can immediately obtain the private key and impersonate me. Moreover, I normally wouldn't take the same precautions with my public key as my private key, but with a compromised-Debian-generated key pair they are practically equivalent. In general, there is no basis to rely on the public key being secret, which you implicitly recommend doing.

Any rational and informed person will immediately regenerate all key pairs generated with a compromised-Debian machine, and remove such public keys from any authorized_keys files in which they appear. They would also do well to consider the Ubuntu strategy of rejecting known weak public keys for public-key authentication.