One Month Later: 300,000 Servers Remain Vulnerable To Heartbleed 60
DavidGilbert99 writes: "The Heartbleed Bug cause widespread panic from internet users around the world worried their sensitive information was being targeted. While system administrators were warned to patch their systems, a security researcher notes that 300,000 servers remain vulnerable to the heartbleed flaw a full month later. He said, 'Last month, I found 1-million systems supporting the "heartbeat" feature (with one third patched). This time, I found 1.5-million systems supporting the "heartbeat" feature, with all but the 300k patched. This implies to me that the first response to the bug was to disable heartbeats, then later when people correctly patched the software, heartbeats were re-enabled. Note that only OpenSSL supports heartbeats, meaning that the vast majority of SSL-supporting servers are based on software other than OpenSSL.' A developer at Vivaldi Technologies AS also pointed out that a significant number of server administrators botched their response, going from safe to vulnerable."
Re:fp (Score:4, Funny)
If this is what you have to do to be happy, my heart bleeds for you...
Re: (Score:1)
You've been waiting to use that line...
We can't patch yet... (Score:2, Funny)
...because we're waiting for vendors to issue patches. Quit whining about the new, latest security hole, because there have been numerous bad security holes in the past, and they will continue to exist in the future.
Re: (Score:2, Insightful)
You should be fired.
Re:We can't patch yet... (Score:5, Informative)
...because we're waiting for vendors to issue patches.
1) who is the "we" you refer to? 9 times out of 10, there are workarounds, ranging from shutting off the heartbeat feature in OpenSSL to parking an SSL proxy host or load-balancer (depending on application) in-between your affected box and the rest of the planet. And yes, it's that fucking important.
2) The "new, latest security hole" in this instance can turn your company's reputation and sales into rancid mush should you get compromised, and in this case, there's no easy way to catch it before they get in. Oh, and don't ask about the potential for lawsuits that a data breach can generate from pissed-off customers.
3) If a vendor hasn't coughed up a fix by now? Stop using the product, and/or learn enough about it to wedge in your own fix until you can replace the product with something whose vendor is more responsive.
4) Sibling isn't entirely flamebait... a competent sysadmin is more than just a keyboard button actuator - he/she should have enough technical mojo to cook up a means to help protect his career and his company in cases like this. If one of my admins told me what you just wrote without providing solid proof that no workaround exists, I'd sit him down and ask him if he really wants to continue his career as a sysadmin.
Re: (Score:3)
3) If a vendor hasn't coughed up a fix by now? Stop using the product, and/or learn enough about it to wedge in your own fix until you can replace the product with something whose vendor is more responsive.
Sometimes it takes more than a month to replace an in place product. Also, just because you're running machines that have OpenSSL on them doesn't mean that anyone in the company has any idea about how to compile code. Right now my father's company is dealing with heartbleed. They take equipment from vendors, and rents the equipment out to customers with servicing contracts. Apparently part of the contracts they have with the vendors is that they're not allowed to do software patches at all. Apparently it wa
Re: (Score:2)
Quit whining about the new, latest security hole, because there have been numerous bad security holes in the past, and they will continue to exist in the future.
Do you apply the same reasoning to war, starvation, and murder?
I've wondered before [slashdot.org]: Does posting as AC make one stupid, or do otherwise intelligent people decide to post stupid things and tick the AC box?
Re: (Score:2)
Then mitigate the issue, or turn your systems off.
Certificate extortion (Score:5, Interesting)
What would help is if there were some certificate system that didn't rely on extortion or exorbitant prices. I know several admins that mitigated the hole but couldn't replace their certificates either because the signer charges a ridiculous revocation fee (I'm looking at you, StartSSL), or because the cost of cutting and signing new certificates was too high. We need a better trust system.
Re: (Score:1)
DANE
Re: (Score:3)
We need a standard for publishing public keys in DNS, then signed by your registrar and secured via DNSSEC (or similar, I'm not claiming to be an expert on exactly what level of security DNSSEC provides). I mean we already trust the DNS system to some extent to handle proof of ownership, extending that system to give some form of cryptographic authentication at the connection level makes perfect sense to me. DNS registration is a weak link in the certificate chain already.
Of course a different name resolut
Re: (Score:3)
It was called DNSSEC Stapled Certificates [google.com]. Was -- Chrome removed it.
See also RFC 6698 [ietf.org].
Note you can already do this with SSH keys, where's the implementation for TLS for HTTP?
Re:Certificates, free: CACERT (Score:1)
Are you new to CAcert?
CAcert.org is a community-driven Certificate Authority that issues certificates to the public at large for free.
CAcert's goal is to promote awareness and education on computer security through the use of encryption, specifically by providing cryptographic certificates. These certificates can be used to digitally sign and encrypt email, authenticate and authorize users connecting to websites and secure data transmission over the internet. Any application that supports the Secure Socket
Re: (Score:1)
I know several admins that mitigated the hole but couldn't replace their certificates either because the signer charges a ridiculous revocation fee (I'm looking at you, StartSSL),
Yup, twenty-five whole dollars. That's the price of several Big Macs, with fries!. Shameless what some CAs will charge.
(Not defending the CA racket here, but $25 isn't really that much when they give the certs out for free. In any case why revoke them, just replace them with a new, free cert. Yes, I know someone can spoof the server using the old cert, but if you want to save the $25...).
Re: (Score:3)
>Yup, twenty-five whole dollars. That's the price of several Big Macs, with fries!. Shameless what some CAs will charge.
(Not defending the CA racket here, but $25 isn't really that much when they give the certs out for free. In any case why revoke them, just replace them with a new, free cert. Yes, I know someone can spoof the server using the old cert, but if you want to save the $25...).
That's $25 per certificate. That may sound cheap to you, but it's not cheap to everyone and especially not when you
Re: (Score:2)
Can't you just get one CA signed certificate and use that to sign all the other certificates in your organization yourself?
Re: (Score:3)
Can't you just get one CA signed certificate and use that to sign all the other certificates in your organization yourself?
Well, until that 'master cert' gets compromised, in which case whoever stole it can immediately turn it into a certificate printing machine...
You sure you want that headache?
Re: (Score:3)
In any case why revoke them, just replace them with a new, free cert.
What is the point in replacing a cert if you aren't going to revoke the old one? Replacing the cert doesn't solve anything if the old one is still valid and usable.
Re: (Score:2)
If $25 per site is a big deal for you, you probably shouldn't be running those websites.
Your complaint is about as retarded is people who bitch about having to pay $200 for an office upgrade every 3 or 4 years while the employee using it makes that much in a day.
If you can't afford to maintain your own websites, stop hosting them and streamline your business until it doesn't suck.
Seriously, $25 per cert? You're doing it wrong if thats even something you think about.
Re: (Score:2)
Selling you something at a higher price than you'd like to pay is not "extortion".
Regardless, issuing certificates involves real work. If you'd like to do that work for free, in perpetuity, eating the costs on your own, then please do! The internet community will love you for it.
Re: (Score:2)
StartSSL won't revoke a certificate unless you pay the $25 revocation fee and they won't just let you cut a new certificate while the old one is unrevoked. How is that not extortion? The only option is to either pay up or find another provider (and leave your old, unrevoked certificate out there).
Re: (Score:3)
If you have a site where an attacker would have bothered with the elaborate process of getting the private key, and then do MITM attacks with it on users, and it would actually matter, you wouldn't have used StartSSL in the first place, and $25 would be absolutely nothing for you.
Hint: not you
Re: (Score:2)
>If you have a site where an attacker would have bothered with the elaborate process of getting the private key, and then do MITM attacks with it on users, and it >would actually matter, you wouldn't have used StartSSL in the first place, and $25 would be absolutely nothing for you.
>Hint: not you
None of which has any bearing on my original point, which is that we need a better and more secure way of applying security to web servers that isn't reliant on the good graces of a third party (either thro
Re: (Score:3)
None of which has any bearing on my original point, which is that we need a better and more secure way of applying security to web servers that isn't reliant on the good graces of a third party (either through their schedule of fees or through their procedures and policies).
I agree on the concept as posted, well sorta. Right now it's a choice between a SSL cert saying "trust me" (self-signed certs), or "trust him" (CA-generated certs.) A better way would be nice, but on a practical level that better way would be damned hard to implement, and I don't even want to know what kind of overhead it would add to the typical user session. I mean, do you really want your local bank handing out RSA keyfobs to each customer just so that you can bank online with a better feeling of securit
Re: (Score:2)
I mean, do you really want your local bank handing out RSA keyfobs to each customer just so that you can bank online with a better feeling of security? From an Ops and Support standpoint, it would be the stuff of nightmares.
You might get away with it for banking, but can you imagine every podunk blog having to do it for each of their visitors? The security people might be happy at the thought, but that would be very short-lived as they'd be at severe risk of having everyone else hang them from the nearest lamppost for being complete asses. Apart from the (reasonable) deaths of some people who have no idea about proportionality, the net effect would be to have no security at all; people will turn off HTTPS before putting up wit
Re: (Score:2)
Actually, unless I'm missing something in TFS, this isn't about rotating your certificate (although that's a good plan if you were vulnerable to Heartbleed, but do your own risk assessment there).
Heartbleed was a vulnerability against openssl, mitigate that and you won't be vulnerable to Heartbleed. You may want to swap out your SSL certs too in case someone grabbed them while you' were vulnerable, but certainly not wanting to pay for the cert rotation shouldn't stop you from updating openssl.
Min
12 of those are mine. (Score:4, Interesting)
We have are 12 dev/test servers that I didn't bother to patch because they'll be decommissioned in a month or two along with the rest of the datacenter they are in, and even though they support SSL connections (with a disposable cert from our private CA), they are generally only used with HTTP and have no private data to protect, and are almost completely unused now.
If someone wants to spend time trying to steal the server's private key or steal user data from the server, that's fine with me, I'd rather have them spend time on my disposable server than someone's real server.
Re: (Score:1)
On a slightly related note, I wonder how many of those are honeypots, either to catch the people using this exploit or to delay the attack on others because they are too busy getting them.
Safe to vulnerable (Score:4, Interesting)
I'm guessing a lot of these are going to be people running older stable distributions that were running versions of OpenSSL from before the bug was introduced. On hearing the news, they may have quickly looked to see if there was an upgrade, seen that there was not, and either installed a newer rpm/deb they found somewhere, or downloaded the latest source they could find (in a source repository for their distro perhaps) and built from source. Meanwhile the patch for the bug had only just been checked into git.
Re: (Score:3)
Then I thought I should gather the version info of all my servers and check those against what is ACTUALLY vulnerable, instead of just upgrading to upgrade.
The SSL problems. (Score:5, Insightful)
I am not a systems administrator (I am a software designer, when I do administration it requires a lot of trail and error.), I do however have to setup an SSL site once every few years. And because of the rarity of this action this is one of those jobs that are difficult to do, compared to other jobs. Sure if your web browser is installed via an Apt-get you are good. However there are times where you need to install it manually, and then you fight and tinker until SSL works, when it does work, your tendency is not to tinker with it anymore.
The issue with Heart Bleed is that it effects open SSL, one of the trouble maker libraries, that require more then just the Basic make config & make & make install.
Now there are a lot of sites setup my armature system admins, many who are less technical then I am, who will get it going and let it run. There isn't any enterprise architecture, the web site is running on a single PC with a single hard drive, chances are the hard drive had already died, and the site is just running from active memory.
Re: (Score:1)
Thank you for this. You sound like one of the folks who get it. There's a lot of push towards devops where developers are actually maintaining the infrastructure. Though many developers can do it, few can do it well. As a current systems administrator (and former developer) I know that it ain't as easy as the devops literature would lead people to believe.
Re: (Score:2)
So you do every IT related job at an expert level?
There isn't anyone who can do all the jobs required in IT and do like a full expert. There are somethings that you don't do often enough and really should get help with... Now if you can't get help, most people can figure out how to do it one way or an other... However the solution may be sub optimal. I setup an SSL website once every 2 or 3 years. Normally I will have access to a Decent Client/Server team who setups the OS and install the software that is
So... (Score:1, Flamebait)
What you're saying is that something has been bleeding for over a month and isn't dead yet?
Does that mean Heartbleed is equivalent to four women?
Browser plug-in (Score:2)
How about a browser plug-in that will stop me from using https if the site is vulnerable? The last thing I want to do is expose my information by passing it through a vulnerable web server. It should be rather easy for a plugin to send a mal-formed heartbeat ping before sending any data to find out if the server is vulnerable, and then block the connection if it is.
Re: (Score:3)
FoxBleed is one. It doesn't block the connection, but it lets you know about the problem.
Re:Browser plug-in (Score:4, Informative)
Thanks! I see there's also one for Chrome:
https://chrome.google.com/webs... [google.com]
I just hope these plugins are trustworthy.
Security Fatigue (Score:2)
Perhaps a lot of server administrators are simply tired of dealing with the unending farce that constitutes modern internet security, and have simply decided to give in. What's the use in spending time and effort on security measures which frequently fail, sometimes spectacularly so in the case of heart-bleed. In particular, what's the point of protecting customer data if organizations like the NSA can simply walk in and take it, or if you're already selling it en-masse to marketers.
Re: (Score:3, Insightful)
I suspect a lot of them
Re: (Score:2)
Because it's the job you're being paid to do? Almost anyone can do the easy parts of server admin. Dealing with tedious stuff like this is the only reason to hire professional server admins.
Let me expose my ignorance... (Score:5, Insightful)
As I understand this, a vulnerable server can expose its private SSL key to an attacker. With this private key, I can decrypt all of its encrypted SSL traffic.
This correct so far?
Now, as I understand this so far, having the private key is great, but I need to be able to MITM the connection to decrypt anything.
How hard is this? At the transport layer, this would require snooping the network connection of the server; someplace locally on the LAN (easiest, port mirror, maybe) or at the ISP (harder, maybe less likely).
The other option would be some kind of DNS spoofing/vulnerability/cache poisoning, redirecting all the server traffic to a system I controlled and then piping it back out. How likely is this?
Re:Let me expose my ignorance... (Score:4, Insightful)
A vulnerable server can expose anything in the server process's memory space to the attacker. Obligatory: http://xkcd.com/1354/ [xkcd.com]
Re:Let me expose my ignorance... (Score:4, Interesting)
As I understand this, a vulnerable server can expose its private SSL key to an attacker. With this private key, I can decrypt all of its encrypted SSL traffic.
As already mentioned, it's anything in the server's memory. Or the client's, since Heartbleed affects clients too.
Now, as I understand this so far, having the private key is great, but I need to be able to MITM the connection to decrypt anything.
It depends whether the connection is using perfect forward secrecy or not. If it's using PFS, then you need an active MITM to grab the session keys, so you can't decrypt old captured traffic and you need to keep your MITM up for new traffic. If there's no PFS, then all traffic ever sent with a given SSL cert can be decrypted with access to that cert's private key. All you need is to passively sniff it, then store it for later on the off-chance you ever get (or crack) the key.
(I'm going to write a small essay on this, because it's important but very poorly documented on the web.)
Given that, you'd think PFS would be common, but according to this study [linshunghuang.com] it's only available on 60-70% of web servers (they don't give a precise number, just 60% that support DHE and 18% that support ECDHE, but those two sets overlap), of which 80% prefer to use cipher suites without PFS, so about half of webservers either don't support PFS or typically won't use it. Slashdot [slashdot.org] doesn't, for example. Neither does microsoft.com [microsoft.com]. I guess that's just the homepage, but then windowsupdate.microsoft.com [microsoft.com] doesn't use it either. It's not supported on outlook.com's web, IMAP, POP3, or SMTP servers. addons.mozilla.org [mozilla.org] and marketplace.firefox.com [firefox.com] also join the club, but their main website and the Firefox update sites do PFS at least. I couldn't find a Google property that didn't do PFS.
And on top of that, of those sites that do use it, 99.3% use 1024-bit DH parameters, which essentially lowers the length of their RSA keys to 1024 bits (which affects the 80% of sites with 2048-bit or longer RSA keys).
If you want to make sure you're actually using PFS, and with decent DH parameters, you generally need to make sure to configure it. Apache does this for you automatically from 2.4.7 onwards (before that, it'll use PFS but only with 1024-bit DH parameters). A lot of other software requires being fed DH parameters manually -- for instance, Courier's IMAP/SMTP servers, ZNC, ircd-hybrid etc. (And when was the last time you configured DH parameters for a server?)
You can check if any given connection supports PFS by looking at the cipher suite in use. If it starts with DHE or ECDHE, it has PFS. (The "E" at the end stands for ephemeral; if it says DH, ECDH, or doesn't mention either of those, then there's no PFS). You can check with e.g. CipherFox [mozilla.org] in Firefox, or using the openssl command-line tools:
$ openssl s_client -connect www.debian.org:443 | grep Cipher
Cipher : DHE-RSA-AES256-GCM-SHA384
If you point it at servers you use regularly, you'll probably be pretty depressed at the results. I know I was when I was making that list above...
Another years old flaw patched (Score:1)
In other news [eweek.com], OpenSSL gets a 4-year-old flaw patched. The catch here is that the bug was not only 4 years in the codebase, but it was publicly reported (CVE-2010-5298 [nist.gov]) for 4 years, without no one taking the responsibility to fix it.
OpenBSD developer Ted Unangst made a detailed report [tedunangst.com] of the bug. It's not as severe as Heartbleed, but still allows remote attackers to inject data across sessions or cause a denial of service (use-after-free and parsing error) via an SSL connection in a multithreaded environme
Need to find vulnerabilities earlier (Score:2)
OpenSSL was actually examined by a lot of tools, but they all missed Heartbleed. My article How to Prevent the next Heartbleed [dwheeler.com] lists approaches that could have found it. We need to improve how we examine this software so problems like this don't happen again.