Millions Experience Browser Problems After Long-Anticipated Expiration of 'Let's Encrypt' Certificate (zdnet.com) 94
"The expiration of a key digital encryption service on Thursday sent major tech companies nationwide scrambling to deal with internet outages that affected millions of online users," reports the Washington Examiner.
The expiring certificate was issued by Let's Encrypt — though ZDNet notes there's been lots of warnings about its pending expiration: Digital Shadows senior cyber threat analyst Sean Nikkel told ZDNet that Let's Encrypt put everyone on notice back in May about the expiration of the Root CA Thursday and offered alternatives and workarounds to ensure that devices would not be affected during the changeover. They have also kept a running forum thread open on this issue with fairly quick responses, Nikkel added.
Thursday night the Washington Examiner describes what happened when the big day arrived: Tech giants — such as Amazon, Google, Microsoft, and Cisco, as well as many smaller tech companies — were still battling with an endless array of issues by the end of the night... At least 2 million people have seen an error message on their phones, computers, or smart gadgets in the past 24 hours detailing some internet connectivity problems due to the certificate issue, according to Scott Helme, an internet security researcher and well-known cybersecurity expert. "So many people have been affected, even if it's only the inconvenience of not being able to visit certain websites or some of their apps not working," Helme said.
"This issue has been going on for many hours, and some companies are only just getting around to fixing it, even big companies with a lot of resources. It's clearly not going smoothly," he added.
There was an expectation before the certificate expired, Helme said, that the problem would be limited to gadgets and devices bought before 2017 that use the Let's Encrypt digital certificate and haven't updated their software. However, many users faced issues on Thursday despite having the most cutting-edge devices and software on hand. Dozens of major tech products and services have been significantly affected by the certificate expiration, such as cloud computing services for Amazon, Google, and Microsoft; IT and cloud security services for Cisco; sellers unable to log in on Shopify; games on RocketLeague; and workflows on Monday.com.
Security researcher Scott Helme also told ZDNet he'd also confirmed issues at many other companies, including Guardian Firewall, Auth0, QuickBooks, and Heroku — but there might be many more beyond that: "For the affected companies, it's not like everything is down, but they're certainly having service issues and have incidents open with staff working to resolve. In many ways, I've been talking about this for over a year since it last happened, but it's a difficult problem to identify. it's like looking for something that could cause a fire: it's really obvious when you can see the smoke...!"
Digital certificates expert Tim Callan added that the popularity of DevOps-friendly architectures like containerization, virtualization and cloud has greatly increased the number of certificates the enterprise needs while radically decreasing their average lifespan. "That means many more expiration events, much more administration time required, and greatly increased risk of a failed renewal," he said.
The expiring certificate was issued by Let's Encrypt — though ZDNet notes there's been lots of warnings about its pending expiration: Digital Shadows senior cyber threat analyst Sean Nikkel told ZDNet that Let's Encrypt put everyone on notice back in May about the expiration of the Root CA Thursday and offered alternatives and workarounds to ensure that devices would not be affected during the changeover. They have also kept a running forum thread open on this issue with fairly quick responses, Nikkel added.
Thursday night the Washington Examiner describes what happened when the big day arrived: Tech giants — such as Amazon, Google, Microsoft, and Cisco, as well as many smaller tech companies — were still battling with an endless array of issues by the end of the night... At least 2 million people have seen an error message on their phones, computers, or smart gadgets in the past 24 hours detailing some internet connectivity problems due to the certificate issue, according to Scott Helme, an internet security researcher and well-known cybersecurity expert. "So many people have been affected, even if it's only the inconvenience of not being able to visit certain websites or some of their apps not working," Helme said.
"This issue has been going on for many hours, and some companies are only just getting around to fixing it, even big companies with a lot of resources. It's clearly not going smoothly," he added.
There was an expectation before the certificate expired, Helme said, that the problem would be limited to gadgets and devices bought before 2017 that use the Let's Encrypt digital certificate and haven't updated their software. However, many users faced issues on Thursday despite having the most cutting-edge devices and software on hand. Dozens of major tech products and services have been significantly affected by the certificate expiration, such as cloud computing services for Amazon, Google, and Microsoft; IT and cloud security services for Cisco; sellers unable to log in on Shopify; games on RocketLeague; and workflows on Monday.com.
Security researcher Scott Helme also told ZDNet he'd also confirmed issues at many other companies, including Guardian Firewall, Auth0, QuickBooks, and Heroku — but there might be many more beyond that: "For the affected companies, it's not like everything is down, but they're certainly having service issues and have incidents open with staff working to resolve. In many ways, I've been talking about this for over a year since it last happened, but it's a difficult problem to identify. it's like looking for something that could cause a fire: it's really obvious when you can see the smoke...!"
Digital certificates expert Tim Callan added that the popularity of DevOps-friendly architectures like containerization, virtualization and cloud has greatly increased the number of certificates the enterprise needs while radically decreasing their average lifespan. "That means many more expiration events, much more administration time required, and greatly increased risk of a failed renewal," he said.
Who is at fault? (Score:5, Insightful)
Regarding my MacBook Air, who is at fault with this error?
The MacOS operating system? Is it supposed to self-update certificates? Do I need to manually install a new certificate?
The Chrome browser?
The Let's Encrypt organization?
My ISP (Comcast)?
The websites I'm visiting? ESPN, Slashdot, etc.?
Re: Who is at fault? (Score:2, Informative)
The WebHost should fix the issue. This is not a user failure or generally even software. Some software I believe has the certificates embedded which is one of the reasons old software is mentioned.
The error your browser gives points to the fault of the host. No update on the browser or your side can resolve the issue...
Re: Who is at fault? (Score:2)
I knew the files were stored locally but I still thought it fell under the webhosts responsibility to provide an updated file?
Re: Who is at fault? (Score:5, Informative)
No. It’s either the responsibility of the OS provider, or the maker of the software you are using to access the site or resource. Websites do not have the ability to install trusted certificates on your computer. That would be a major security flaw.
Re: Who is at fault? (Score:5, Informative)
Web hosts provide only the End certificate for the website itself, and any Intermediary signing Certificates; They do Not generally include the self-signed Root CA certificate, and in general... the Web host cannot cause your browser to trust the root Cert.
The web host MAY sometimes provide a root certificate cross-signed by another root CA, and that would work --- but they would have needed LetsEncrypt to provide such a certificate in the signing chain to be included.
In this case the DST Root Certificate expired which had cross-signed the LetsEncrypt ISRG X1 root, And there is No new Cross-Signed copy of the X1 cert in existence - only the Self-Signed one.
For your browser to recognize the LetsEncrypt-issued certs as valid... The browser Must have the Self-Signed version of the X1 Certificate installed and trust and rely upon that signing chain, instead of the Cross-Signed version of the Root certificate.
In addition: If the web host is presenting Intermediary certificates chained to that Cross-Signed version instead, Then authentication will fail... To avoid issues, they had to Delete their intermediary certificates and include only the Current LetsEncrypt intermediate certificate chain based on the Self-Signed X1 certificate. If your browser does not Trust the Self-Signed X1 certificate - due to running an older OS version such as MacOS Mavericks which no longer receives Certificate updates, Then you will still have Cert errors which the end user cannot fix (Other than by manually installing the new roots on their own).
Re: Who is at fault? (Score:2)
Thanks for the detailed explanation.
Sounds like a heinous standard from a user experience sense but it's clear my understanding was flawed, so I likely do not understand the security risks associated with better distribution of the root CAs. Seems weird to me the blame falls on the OS.
Re: Who is at fault? (Score:5, Insightful)
No, the blame falls on the self-important security-uber-alles assclowns who are obsessed with the idea that a single point of failure, one with a short expiration date no less, is just what the Internet needed.
Re: Who is at fault? (Score:2)
Right. I mean the immediate blame for the fix is the OS but the standard seems terribly fault. It would be interesting to hear alternative implementations.
Re: (Score:1)
No, the reverse of that.
It is a really good thing. You can choose which certificates you trust and which you distrust. The organizations can recommend that you don't trust the certificate after a date which is a good security practice, but still ultimately up to the user to determine if you trust expired certificates or not. It is amazing that you as the user have the choice, unless forbidden by a tyrannical school, employer, or government that removes the choice on your machine.
It isn't just the date. If
Re: Who is at fault? (Score:5, Insightful)
You can choose which certificates you trust and which you distrust. The organizations can recommend that you don't trust the certificate after a date which is a good security practice, but still ultimately up to the user to determine if you trust expired certificates or not
Consumers don't care about this "freedom". At all. They primarily want to use whatever websites they need, and secondarily they might be vaguely thinking that it would be nice if their communications were secure. The intricate nuances of certificates are not on that list.
The whole certificate architecture is, IMHO, broken. Putting certificates with limited lifespans onto embedded devices or devices with no hope of ever receiving an update is a sure-fire recipe to increase the flow of devices headed to landfills.
Re: (Score:3)
Yep. It should be telling that the first thing ordinary Joe does when presented a certificate error is to try to ignore it. It's so prevalent that Chrome removed the ability to do so in order to avoid the entire security scheme from falling apart.
For a non-technical person, their troubleshooting goes something like this:
1. Try again
2. Click on random things in the hope that it works
3. Google the problem (but give up because of the incomprehensible jargon)
4. Try again later
5. Try again from somewhere else
6.
Re: (Score:2)
I am a very experienced sysadmin and I dont have the time to deal with this crap at all. Never mind the users
This nerd attitude of blaming users who just want to use the stuff and have it work is really annoying.
Re: (Score:2)
+1 to that.
A few years ago I helped my folks get a security camera system, like Nest cameras but older and from a traditional home security company. It came with a flyer saying "do not click the SSL checkbox on the admin page". No explanation why. I did click on the checkbox of course, and then ... I had to download a version of Firefox from years ago that supported the SSL version of the security system. Nice, the workaround for broken encryption was to use zero encryption ... not sure how that helps. The
Re: Who is at fault? (Score:2)
Re: (Score:2)
Sounds like a heinous standard from a user experience sense but it's clear my understanding was flawed, so I likely do not understand the security risks associated with better distribution of the root CAs. Seems weird to me the blame falls on the OS.
Unfortunately, it is a mix of people wanting security systems that "just work", and the fact that security systems require constant maintenance to remain secure.
While people try to pin blame on parts of the system, the root of the problem is that security requires constant maintenance. You cannot fail to maintain the system and still expect it to remain secure. The crux of this problem is devices and software that still use stale certificates that were replaced back in 2017.
Open a device that hasn't be
Re: (Score:2)
I guess what I am asking for is a piece of software that allows you to update even on a deprecated system. The fact that this type of software is not inherently designed into modern web-browsers seems a bit of a shortcoming regarding the security standard as a vital part of internet infrastructure.
Again, security is an area of significant weakness for me but it seems there could be something like PGP that allows users to authenticate to a server and likely even validate the certificate against multiple serv
Re: (Score:2)
In this case the DST Root Certificate expired which had cross-signed the LetsEncrypt ISRG X1 root, And there is No new Cross-Signed copy of the X1 cert in existence - only the Self-Signed one.
For your browser to recognize the LetsEncrypt-issued certs as valid... The browser Must have the Self-Signed version of the X1 Certificate installed and trust and rely upon that signing chain, instead of the Cross-Signed version of the Root certificate.
Though this may have led to connectivity issues for some, it makes it sounds as if the root CA issuance or distribution is at fault. I would instead argue that the behavior you described is a bug in the browser or its TLS engine. Both the self-signed and cross-signed versions of ISRG Root X1 have the same public key (see below), making them cryptographically identical. What policy reason would there be for not recognizing them as the same, and succeed in validating the certificate chain once a trusted ancho
Re: (Score:2)
What policy reason would there be for not recognizing them as the same, and succeed in validating the certificate chain once a trusted anchor was reached
Well; the possible observation is about the certificate handling in Chrome on OS X and also the Windows operating system, at least so far as 2012. I'm not so sure this can be considered a bug (they are doing what the RFCs say they should do most likely) rather than a common misconfiguration and poor instructions from the certificate authorities or lack
Re: (Score:2)
The root of trust is in the browser. (or OS) Otherwise any web host could just present a new root certificate to stake their claim to be secure and trustworthy.
Re: Who is at fault? (Score:2)
Re: (Score:1)
If the devices were Ivanti (formerly Pulse Secure, formerly Juniper) Pulse Secure vpn devices, the old expired R3 CA certificate had to be removed from the intermediate certificate store before the new R3 CA would be used.
Re: (Score:2)
Yes... Also, rather, there was a version of the R3 certificate that was Not expired yet BUT Certificates that had signed that cert expired on September 30th.
Which is a major problem in that the Server may have thought that cert to be valid enough to continue to present to clients, and the clients not.
Re: Who is at fault? (Score:5, Informative)
Letsencrypt brought short lived certificates to the web. Their rise coincided with widespread use of https for websites that didn't need high security, so their use is widespread. They weren't initially in the trusted root lists of browsers and other software, so their root was signed by other providers to make it trusted in existing browsers. It is one of those other providers' certificates that expired on September,30th, and the other one they use that is OK until 2023 only started appearing in trusted root lists in 2017. Their own root certificate came a bit later to trusted root lists.
Re: (Score:2)
Re: (Score:2)
This caused an issue with OpenBSD, a patch was released on the 30th to address the problem
https://ftp.openbsd.org/pub/Op... [openbsd.org]
Before the patch I had minor issues accessing IMAP email, after I applied the patch, all good. Other people had issues trying to down load the patch.
Re: (Score:1)
Re:Who is at fault? (Score:4, Informative)
Assuming you have kept the OS up-to-date, it sounds like a bug in the TLS library used by the browser. It doesn't seem to handle cross-signed root certificates correctly.
Let's Encrypt's root certificate (also called ISRG) was cross-signed by IdenTrust. This is a common industry practice to help establish new roots. The expiration of IdenTrust's root shouldn't have been an issue because your OS or browser likely trusts Let's Encrypt's cross-signed root as well, but it seems that the TLS library used tries to validate the entire certificate chain, not just the certificate chain from the first known trusted root Let's Encrypt.
Re: (Score:2)
In many cases there are no options for keeping the OS up to date. My company made an Android device back in the Android 2.3 days that still has a few thousand devices out in the wild. Hardware vendors are unwilling to support drivers for upgrading to a more recent version, and early versions of Android 4 were less stable than Android 2.3, until 4.4 became the next stable point with lots of devices stuck on it as well, but that required a dual core SoC for the UI performance improvements, so single core d
Re: (Score:2)
Why would you have outdated and unsupported hardware connected to the internet? Do you also go an a wild sex spree with south African prostitutes and no condom in sight? Do you chose to not vaccinate against COVID and run around licking handrails during the pandemic?
Re: Who is at fault? (Score:2)
This is why so many industrial control system PLCs have no security. Because f* you if I have to rip out a few million dollars worth of equipment every few years. Just so some tweens can keep posting selfies to some college perverts "Hot Chicks on Campus" web site.
Re: (Score:2)
Because there's the rare software that needs Windows 9x, and most users want to install basic software on that OS for it to be usable.
When I'm running this special case, I know what I'm doing. I'd rather be able to do what I need, rather than have to troubleshoot what should be a basic protocol and struggle by finding a workaround.
Also, since forever, most outdated hardware and software can be put behind a firewall, which is more
Re:Who is at fault? (Score:5, Insightful)
Who's at fault? The protocol/implementation designers.
The original intent of that protocol was to say that it's end-to-end secure with no unexpected MITM, with the browser complaining should there be a missing certificate.
Let's Encrypt, HTTPS Everywhere, and similar projects want everything encrypted. Consequently, every site needs a certificate pre-signed by Lets Encrypt. If they decide to shut down, that cascades into many other sites not having a valid certificate.
There was some project to do a "verified secure" with a green lock symbol, to distinguish really trusted certificates rather than just random HTTPS. Google decided to drop that.
And after all that, there's plenty of websites that rely on HTTPs as just encryption (satisfiying most user's intent) instead of authentication (which is expected by the protocol) - for something that was presented to the user as "people can't see what you submit on this form" back in the Netscape 2.0 era.
Re:Who is at fault? (Score:5, Insightful)
Yup. This is failure by design. The Web PKI is designed around the concept of certificates expiring and expiring and expiring in order to ensure a continuous revenue stream for the gatekeepers, the CAs. This isn't an implementation issue or a technical problem, it's this way because it was designed to be this way. In particular for any kind of IoT, which is supposed to run unattended forever, a TLS-enblement mechanism that shuts itself down every twelve months (end-entity certs) or every few years or even decades (CA certs) is totally, utterly wrong.
However, the fact that it's totally unsuited to the task won't stop the Web PKI from being pushed into everything anyway.
Re: Who is at fault? (Score:2)
There's nothing stopping you from generating a cert with a 100 year expiry for these situations.
The problem seems to me to be that browsers reject certs older than a year.
Re: (Score:3)
Chrome uses the certificate store of the host operating system, which means your Mac OS is where the problem lies. Unless Apple pushes out updates to your machine that include new root certificates, your browser won't recognize the chain of trust. I don't really know Mac OS, but I suspect that you can install your own certificates on it, as is common in all TLS stacks I have worked with (windows, openssl, java, etc) so you can just manually add the needed root certificate.
Re: (Score:2)
MacOS Sierra was end-of-life in 2019.
This means Apple has not been updating the associated certificates... since 2019.
EOL OS = no updates for root/intermediate certs. If you want to use a device that is out of support, expect bad things to happen. Expiring certificates is by far the most benign outcome...
Re: (Score:1, Flamebait)
This is directly the fault of Google, Microsoft and Firefox for jumping on the HTTPS bandwagon with their browsers and not supporting the entirety of H2 (namely the unencrypted mode.)
Thus H2 was held hostage by requiring all sites to be also be HTTPS.
Then Google made it worse by making all ads have to be HTTPS. All other ad vendors followed suit.
Then Google made it even worse by labeled non-HTTPS sites as dangerous.
Users should have always had the option to ignore the certificate or use the non-HTTPS versio
Re: Who is at fault? (Score:2)
Re: Who is at fault? (Score:2)
Re: (Score:2)
I dealt with this on my MB Air and Pro running El Capitan, you just need to download and import the certificate into Keychain and you'll be right.
Chrome uses the system's certificate store, whereas Firefox uses its own - I didn't realise that at first and was really confused about why one worked and not the other. But the fix for you is actually really really simple, download the cert, add it to keychain, and set it to trusted. You download it from letsencrypt: letsencrypt dot org slash certs slash isrgroo
Re:Who is at fault? YOU, and that is GOOD. (Score:2)
The answer is difficult to hear, but simply: YOU are at fault. And it's a good thing.
Your operating system provides a bunch of pre-installed certificates, saying basically "We trust these certificates, and provide them hoping they are useful." You are free to remove any of them or mark them as untrusted, you are free to add to them and make any other certificate marked as trusted. They will update certificates that are still trusted as a convenience for the bulk of the users, but ultimately it is not the
Re: (Score:2)
who is at fault with this error?
The TLS protocol, for not allowing the use of multiple certificate authorities like GPG does.
If we could all sign our sites with multiple authorities, it would not be a major problem if one has an issue.
Follow the bouncing expiration date (Score:4, Interesting)
Digital certificates expert Tim Callan added that the popularity of DevOps-friendly architectures like containerization, virtualization and cloud has greatly increased the number of certificates the enterprise needs while radically decreasing their average lifespan. "That means many more expiration events, much more administration time required, and greatly increased risk of a failed renewal," he said.
I had to renew a certificated in August that was quite the nightmare. The certificate itself was issued by an enterprise that has a contract with a certificate authority to act as an intermediary. However, they clicked on the wrong button when issuing this one, and issued an internal certificate instead of a public-facing one, and it took almost a month for anyone to figure out why it didn't work.
At the end of the process, I had a talk with someone who is involved with this professionally, like Mr. Callan, and I was warned to expect greater havoc as Apple and Google push harder to make ALL certificates last 100 days or less, like Let's Encrypt.
Fortunately, the process I've been using for a decade is possible to automate, as long as no one screws things up like they did in August.
Re: (Score:3)
Seems the heart of the matter should be certificate management, after all certificates aren't going away, and if they're going to be expiring even faster, that'll just make it all the more important.
Re: (Score:2)
We need a "certificate alert" state where applications will start to warn Less-Disruptively when any cert they are relying upon is 48 hours from expiration, and also for Intermediary signing certs: A URL attribute for clients and servers to automatically attempt to retrieve a new version, when it's close to expiration.
Certificate expiration only matters to sysadmins. (Score:5, Interesting)
Certificate expiration really only matters to sysadmins. End users don't know what to do or who to tell if something is expiring.
To solve this problem, I wrote a python script that iterates over the set of certificates that have been issued to my organization (easily available from the CA) and attempts to https handshake with all of the CNs and SANs associated with certificates that are approaching expiration. This runs as a recurring jenkins job that notifies my team for anything that replies with a certificate that is expiring soon.
This has saved my ass several dozen times since I inherited responsibility for a couple hundred random virtual hosts along with my current position, some of which were legacy services that nobody still employed knew about until the alert popped up.
The real problem is oddball/backend services that use TLS but don't answer incoming https connections on 443-- https handshakes don't find those.
Re: (Score:2)
To solve this problem, I wrote a python script that iterates over the set of certificates that have been issued to my organization (easily available from the CA) and attempts to https handshake with all of the CNs and SANs associated with certificates that are approaching expiration. This runs as a recurring jenkins job that notifies my team for anything that replies with a certificate that is expiring soon.
This has saved my ass several dozen times since I inherited responsibility for a couple hundred random virtual hosts along with my current position, some of which were legacy services that nobody still employed knew about until the alert popped up.
The real problem is oddball/backend services that use TLS but don't answer incoming https connections on 443-- https handshakes don't find those.
Ninja & Ghost Certificate Source Tracking
Good job writing an automation process to pull and check the certificates of your endpoints issued by your CA. You're lucky to have been able to enumerate the certificates issued by your CA. I wasn't so lucky...
In the organization that I work for currently we had a similar problem but a much worse one. We had outage events a few times a year due to certificate expiration on various servers and also applications due to certificates expiring, except that these
Re: (Score:2)
It matters to end users in the sense that the faster they expire the less likely the end user is to get ripped off.
Re: (Score:2)
...I had a talk with someone who is involved with this professionally, like Mr. Callan, and I was warned to expect greater havoc as Apple and Google push harder to make ALL certificates last 100 days or less, like Let's Encrypt.
That actually might improve things. If all certificates only last 100 days, then renewing them will become routine for both users and CAs. That should help reduce the errors over time as people get used to the process and failure points are found and fixed.
Re: (Score:1)
This issue is not caused by normal certificates that expire quickly. It's because a root CA was totally deprecated. In other words, no new cert was issued for the root when it expired. This is a big deal.
This doesn't happen very often (like never) so it's something a lot of software doesn't expect.
Arguably this is partly Let's Encrypt's fault for not using a proper root CA then pushing it out to software vendors. They wanted to jump-start their system by using a CA already accepted by most software and this
Re: (Score:2)
Re: (Score:2)
Yes, several reasons. But I don't see where that is relevant to the conversation.
Re: (Score:1)
Curious - I can't find any evidence of Apple or Google lobbying for an even further certificate expiration shortened timeline. Do you happen to have any links/references you could share?
Explain it to non-technical users (Score:2)
Re: (Score:2)
Seriously though, I'm of average "technical" ability and my eyes gloss over when it comes to SSL, root certificates, and all the rest of it. Explain to a technical person who doesn't spend time with SSL implementation...
Re: (Score:2)
Shouldn't one be able to search for the old certificate, and replace it if needed?
Re: (Score:2)
There's plenty of blame to go around. Starting with the whole certificate system being designed to be fragile and depend on "Authorities" to work. Then there's the browsers for insisting on shorter and shorter lived certificates and refusing to implement pinning even though it would make things more secure. Also for forgetting that the software is supposed to do what I say, not the other way around. Then the OS and firmware vendors for not providing a timely update and for not making it simple for the user
Re: (Score:3)
Also for forgetting that the software is supposed to do what I say, not the other way around.
Yep. The newest versions of Chrome no longer have a button to let you override and go to site with an expired certificate. The certificate's broken? You're not going there even if it's just pictures of kittens.
Re: (Score:2)
Originally I felt it was more on users who haven't updated their OS but we had reports of users on newer OS versions having this issue.
The previous CA who signed LetsEncrypt's root was totally deprecated - making that version of the ISRG X1 Root certificate invalid.
Web hosts needed to erase all copies of Intermediary certificates and Root certificates from the Previous CA, and Install the new copy of the intermediate signing certs signed by the SELF-Signed version of the ISRG X1 Root certificate. Pr
Re: (Score:2)
Yeah but Let's Encrypt is still providing the old root in their certificate bundles because they think it'll make old Android devices keep working.
And apparently there are a lot of people running Windows with updates turned off, or Macs they haven't updated since 2016. Christ this has been a pain.
Re:Explain it to non-technical users (Score:5, Informative)
My best attempt at a low-tech explanation for this would be to step through how certificate chain validation works, and why it failed in this specific case.
When a web browser visits a site like slashdot.org that uses the default certificate strategy from Let's Encrypt, this is how the browser will fail validating its valid certificate:
1. The website send us its leaf certificate for slashdot.org, signed by another certificate calling itself R3. This leaf certificate hasn't expired, but we have no reason to trust it, so let's try to validate the certificate for R3
2. The website also sent us the certificate R3, signed by another certificate ISRG Root X1. This R3 hasn't expired, but we have no reason to trust it either, so let's try to validate ISRG Root X1
3. We were not given the certificate ISRG Root X1, but my OS in 2017 had an update that added this certificate in a store of trusted certificates. It was also signed by another certificate DST Root CA X3 to speed up its adoption. This ISRG Root X1 hasn't expired, and my OS trust it, so I ought to trust slashdot.org, but let's for some reason try to validate DST Root CA X3 as well
4. We were not given the certificate DST Root CA X3, but my OS came with this certificate pre-installed, It expired on 2021-09-30 though, so I can't use this for validating anything.
So by step 3 we knew were done. Why did we keep looking for certificates after that? Software developers of TLS libraries used to wonder the same thing, and fixed this issue a few years ago in their code. Not everything hooked up to the Internet runs updated software though, which is why Let's Encrypt tried to warn [letsencrypt.org] developers in January about upgrading affected TLS libraries:
* OpenSSL < 1.1.0
* LibreSSL < 3.2.0
* GnuTLS < 3.6.14
Re: (Score:2)
This is exactly it, ambiguity with cross-signing verification that wasn't resolved until recent releases of SSL/TLS libraries.
Re: (Score:3)
`openssl verify`, however (even openssl post 1.1.0), will still fail to verify a server cert with a provided fullchain.pem without the -partial_chain or -trusted_first option.
LibreSSL (all versions) `openssl verify` is totally worthless, and can never be convinced to do the right thing.
The other option is to issue your certs with certbot --preferred-chain to tell certbot to omit the ISRG Root X1 cert, which says it was issued by DST Root CA X3, and only include the R3 cert.
Re: (Score:2)
`openssl verify`, however (even openssl post 1.1.0), will still fail to verify a server cert with a provided fullchain.pem without the -partial_chain or -trusted_first option.
Hmm, are you sure about this? I'm not aware of the command for checking an entire chain file at once, but I've learnt to always use this command:
$ openssl verify -verbose -no-CApath -no-CAfile -CAfile <(cat isrg-root-x1.pem dst-root-x3.pem) -untrusted r3.pem slashdot.org.pem
slashdot.org.pem: OK
$ openssl version -a
OpenSSL 1.1.1f 31 Mar 2020
built on: Mon Aug 23 17:02:39 2021 UTC
platform: debian-amd64
Re: (Score:2)
yes, -untrusted along with -CAfile works, but defeats the purpose of openssl verify, if i'm understanding your usage correctly, since it can't be used as a general purpose verification tool, for example, in CI or monitoring where not all certs are LE.
It would be more useful if `openssl verify -CAfile fullchain.pem cert.pem` always worked, regardless of the cert, by assuming fullchain.pem was self-contained (such that it could also be used to check entirely self signed, non public CA signed chains)
Re: (Score:1)
Only dreamers think that manufacturers will provide an update before the certificates expires for the computers or other devices bought this year.
Tic tac tic tac tic tac
Broken by design [TM]
Re: (Score:2)
Hell, I’m a technical user and did not really understand what to do to maintain service continuity. For a system that provides so little real security, it sure is a mess.
Re: (Score:2)
"When this happens, add a permanent exception and move on." If the system won't get fixed until it's truly broken, you have to break it some more.
Re: (Score:2)
They changed the guard at the door and you have to teach the new guard who you are or they won't let your visitors in because the guard doesn't know who they have an appointment with.
Re: Explain it to non-technical users (Score:2)
And as usual, nothing will be fixed (Score:3)
We keep having these examples of the dangers of the growing centralization of the Internet, but a month from now, I'm sure absolutely nothing will have done to prevent this sort of thing from ever happening again. There will still only be 2-3 major hosting companies, 2-3 major CDNs, and so on, until there's another merger season, then we may be left with only one major player in a given category.
Re: (Score:3)
This particular issue is very distantly related to centralization though. I would rather suggest these more specific issues:
1. TLS libraries have bugs [letsencrypt.org] in how they validate certificates. People will keep hooking up software with outdated dependencies to the Internet as long as it's possible to.
2. Certificates that end up being pinned or hard-coded by software often have a ludicrously long lifespan of 15-30 years. By the time one such certificate nears expiration people either weren't expecting it, or are unf
Re: (Score:2)
We keep having these examples of the dangers of the growing centralization of the Internet
You realise that this entire thing occurred as the result of adding an additional trusted root right? This is very literally an example of *decentralisation*. Let'sEncrypt now no longer relying on IdenTrust.
I'm sure absolutely nothing will have done to prevent this sort of thing from ever happening again.
I hope we see this issue come up again. Because I don't like centralisation and we should not be reliant on just a few root authorities.
There will still only be 2-3 major hosting companies
What's a hosting company? Is that a thing people use when they don't know how to internet themselves?
I do agree with you on CDNs, but until you fork out the literal bill
Re: (Score:2)
What's a hosting company? Is that a thing people use when they don't know how to internet themselves?
Or when ISPs make it impractical for a subscriber to internet themself, either because of the IPv4 address shortage or just to make inbound ports an upsell.
Weird certificate caching (Score:3)
On one site I look after running Exchange with LetsEncrypt certificates, the certificate is updated every month and had been migrated to the new certificate chain several months ago. For 9 out of 10 users, there was no impact, but for one iPhone X user, the certificate error popped up. Despite updating the phone to the latest OS and ensuring all apps were up to date, the problem persisted. The solution was to delete the account from the phone and add it back.
Strange thing is, two others users had the same phone, and did not experience a problem. It appeared to me the phone in question had cached an older certificate and was trying to use that. You would think the client would check for an updated certificate on every connection attempt, but apparently not in some cases.
Workarounds ... (Score:2)
I'm guessing "Let's Decrypt" wasn't one of them.
Not my problem (Score:2)
I'm handling 100s of certificate of Let's Encrypt (Score:2)
Re: (Score:2)
My "outside of the box" view tell me that the WHOLE CONCEPT OF EXPIRATION is pretty ridiculous.
It's pretty important for end-entity certs, because the system allowing certs to be revoked is majorly limited - and use of short-term expiration is a much more effective less-resource intensive (on the CAs' resources) manner of making sure retired certificates are invalidated.
Expiration exists mainly for important security reasons as issuance policies and minimum key lengths, etc needed for strong authenticity,
Re: (Score:2)
If it provided real security, sure. Most of the certs today are just compliance certs— the functional role they play is trivial. There is no good reason for my internal home automation system to need a new cert every 90 days.
Re: (Score:2)
Re: (Score:2)
Erh... that's a terrible idea.
The whole "at some point companies discontinue products" is the whole reason for that experiation date. Because past that discontinue date, they don't give a fuck about their product anymore. Including the certificate they use. Now imagine that certificate getting into the wrong hands.
That experiation date at least ensures that at SOME point in time, even a hacked certificate loses its ability to cause damage.
If anything, allow people to explicitly ignore expiration dates. If t
Re: (Score:2)
Re: (Score:2)
Yes. Maybe that way people will learn not to buy that crap.
Stop certificate pinning (Score:2)
Good. If it hurts, do it more frequently, and bring the pain forward.
A significant part of why this root ce
No problems here (Score:1)
The expiration of a key digital encryption service on Thursday sent major tech companies nationwide scrambling to deal with internet outages that affected millions of online users
Huh, I honestly had not noticed, have seen no issues so f#&*$3*#$!& NO CARRIER
Only Microsoft did this well (Score:2)
We've graduated (Score:2)
Before, we only had link rot taking down pages off the internet. Now we get to see whole domains going dark when certificates expire.
Re: (Score:2)
Whole domains go dark when hosting plans expire and when domain name registrations expire.
Why didn't they fix it beforehand ? (Score:1)