Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 



Forgot your password?
typodupeerror
×
Security The Internet

DigiCert Revoking Certs With Less Than 24 Hours Notice (digicert.com) 61

In an incident report today, DigiCert says it discovered that some CNAME-based validations did not include the required underscore prefix, affecting about 0.4% of their domain validations. According to CA/Browser Forum (CABF) rules, certificates with validation issues must be revoked within 24 hours, prompting DigiCert to take immediate action. DigiCert says impacted customers "have been notified." New submitter jdastrup first shared the news, writing: Due to a mistake going back years that has recently been discovered, DigiCert is required by the CABF to revoke any certificate that used the improper Domain Control Validation (DCV) CNAME record in 24 hours. This could literally be thousands of SSL certs. This could take a lot of time and potentially cause outages worldwide starting July 30 at 19:30 UTC. Be prepared for a long night of cert renewals. DigiCert support line is completely jammed.
This discussion has been archived. No new comments can be posted.

DigiCert Revoking Certs With Less Than 24 Hours Notice

Comments Filter:
  • Is this going to be like Crowdstrike, but with webservers?

    • Probably not. There's much more competition in the digital certificate market and DigiCert's slice of it is miniscule.

      • Thanks - I certainly hope so. :-)

        • by ls671 ( 1122017 )

          More like secure boot. PKI is kind of going to hell lately because of all the bean counters who modified the principles on how to apply it properly and took all kinds of silly shortcuts thinking; nobody will ever notice the difference.

          As an early adopter of RSA in ~1980:
          https://en.wikipedia.org/wiki/... [wikipedia.org]

          I kind of saw that coming ~2000 since same shortcuts were taken in projects I was working on, despite the architect team wishes.

          "In a public-key cryptosystem, the encryption key is public and distinct from th

      • Worth noting that Crowdstrike has plenty of competition and their market share is also miniscule. The question is, whether their market share happens to include something critical that affects a lot of people (like say... Airlines). I suspect if flights weren't affected by Crowdstrike it would have been just as newsworthy as Crowdstrike's previous issues (which were reported only in tech rags rather than every news agency on the planet).

        • by DarkOx ( 621550 )

          Crowdstike has something like 80% of the F500. The market share might be small if you define the market place as all EDR/XDR products, but if you look at EDR/XDR for the large enterprise, CS is approaching natural monopoly status.

          • Crowdstike has something like 80% of the F500. The market share might be small if you define the market place as all EDR/XDR products, but if you look at EDR/XDR for the large enterprise, CS is approaching natural monopoly status.

            This brings an interesting discussion. We mostly look at DR/BCP risk and mitigation within a single corporation or enterprise, but should we be considering the aggregate risk and impact with the entire F500 being an entity to evaluate risk? What is the greater impact if F500 as a whole is impacted by an outage?

            Maybe these are the kinds of discussions that need to happen in order to open up the desktop to something other than Microsoft and Microsoft. Which is THE eggs-in-one-basket problem.

      • Also, with an expired/revoked cert the user can (hopefully) say "this is clearly bogus" and continue anyway (Firefox is one notable exception that springs to mind, "the i in 'certificate' isn't dotted correctly, we won't allow you to continue"). With Clownstrike they didn't have that option.
    • by gweihir ( 88907 )

      No. The numbers are not comparable and machines are still accessible. Also, it is pretty easy to get a cert from another source.

    • by AmiMoJo ( 196126 )

      Depends how the certs were distributed. Ideally the update should be automatic.

  • by Sloppy ( 14984 ) on Monday July 29, 2024 @10:08PM (#64665692) Homepage Journal

    Well, we've known for over 30 years that having multiple parties attest to a key's owner's identity is really easy (PGP could do it) so I'm sure one little provider's problem won't cause anything widespread. Surely a modern, fundamental protocol like HTTPS doesn't rely on anything with quaint limitations.

    • by ctilsie242 ( 4841247 ) on Tuesday July 30, 2024 @12:08AM (#64665804)

      This has been a head-scratcher for me since SSL was invented. Why does it have just one root key, and no more? Why can't I get a certificate for my TLS terminator (be it a reverse proxy, load balancer, or the actual program) that is signed by multiple vendors? This way, if one vendor has issues, it will still work. This redundancy would save businesses a ton of time and man-hours. Especially if something happens and someone forgets to renew.

      We have redundant subsystems for everything else... but only one root CA per SSL/TLS cert.

      This also will make life easier, as I could have the cert renewed often via LE/ACME, but have it signed by a commercial company, perhaps with extra vetting for an EV certificate to show that the web server is working.

      Of course, what also would be nice is better availability of HSMs, so I can generate the private key for the webserver on a PCIe card, perhaps back it up or sync it among HSMs using m of n security procedures, which ensures the bad guys can't just copy off the public key.

      • by beezly ( 197427 ) on Tuesday July 30, 2024 @03:13AM (#64666064)

        An interesting idea. Cross-signing sort of achieves this.

        How would certificate revocation work with multiple signers? If one signer said the certificate was revoked, but the other said it wasn't, which would you trust?

        • That is where things get hairy. In GPG, one can set a threshold. For this, if one revocation cert is issued, the key should be marked invalid. If a CA is sending out fake revocation certs without proper authorization, it gets sacked, just like a CA that issues broken or bogus certificates.

          • by dgatwood ( 11270 )

            That is where things get hairy. In GPG, one can set a threshold. For this, if one revocation cert is issued, the key should be marked invalid. If a CA is sending out fake revocation certs without proper authorization, it gets sacked, just like a CA that issues broken or bogus certificates.

            That defeats the whole purpose of multiple signers, though. In this particular case, for example, the validation process itself was flawed, and DigiCert sent out revocation without the customer's authorization. So DigiCert should be sacked (and I would argue that for the affected customers, that should absolutely be the outcome), but that doesn't avert the crisis.

            IMO, the best answer is probably a plurality of responding/reachable certificate authorities, erring towards validity in the case of an exact ti

        • by Sloppy ( 14984 )

          If one signer says their certification is still valid, and if you trust that signer (as people always completely do with X.509; there's no such concept as "moderately trusted") then it's a valid cert.

          That someone else revoked it, doesn't mean anything is wrong with it. For whatever reason, they're just no longer attesting that it's correct. not-known-to-be-correct != incorrect. As long as someone you trust says it's correct, then you should believe it's correct. (Either that, or rethink how much you trust t

          • That is a bit problematic though. Normally if a certificate is revoked this is because it should not be trusted, period - it was cracked, compromised, fraudulent, etc. Like a counterfeit bill, if one teller says it is fake but the teller to the left and right say it's good, then the default should still be to doubt it.

            • by dgatwood ( 11270 )

              That is a bit problematic though. Normally if a certificate is revoked this is because it should not be trusted, period - it was cracked, compromised, fraudulent, etc. Like a counterfeit bill, if one teller says it is fake but the teller to the left and right say it's good, then the default should still be to doubt it.

              Not necessarily. In this case, if I understand correctly, these certs are being revoked because their standards weren't followed correctly in constructing temporary DNS records for the purpose of verifying the identity of the company requesting the certs. And the details of that validation are specific to DigiCert, so that would not affect the validity of any other certificate authority's validation.

              So DigiCert isn't saying that the certs *can't* be trusted, but rather that DigiCert can no longer vouch fo

              • But do the browsers know this? Does the browser or certificate authority distinguish "revoked due to legalistic reasons" versus "revoked because something's wrong with it", and take different actions in the code? Or does the user still have to manually click on the "trust this cert anyway?"

                • by dgatwood ( 11270 )

                  But do the browsers know this? Does the browser or certificate authority distinguish "revoked due to legalistic reasons" versus "revoked because something's wrong with it", and take different actions in the code? Or does the user still have to manually click on the "trust this cert anyway?"

                  Depends on how they marked it in the revocation list. I'm going to guess "privilegeWithdrawn". That can include such vague things as "CA terms of service violation", and is the closest to a plausible revocation reason code that could reasonably still allow trust from other signers..

      • by unrtst ( 777550 )

        Why does it have just one root key, and no more?

        You're likely well aware of this already, but just to clarify - your cert has but one root cert, but there are multiple root certs out there. PGP/GPG rely on numbers of people trusting a cert to built a web of trust with implied authority. SSL/TLS certs rely on established authorities, and all users keep a list of those they trust, which is an inherently different structure.

        Why can't I get a certificate for my TLS terminator ... that is signed by multiple vendors? This way, if one vendor has issues, it will still work.

        IMHO, there is an easy way to get a very similar result - obtain multiple certs for each site, each from a different vendor. IE: rather

      • Cross signing is real, just have to convince (possibly your enemy) them to do that for you. This has been done in the past (I know of a case from less than 8 years ago or so).
      • There are indeed methods and patends to do this. Ie, authentication involving N parties with mutual distrust. I cited something similar in a patent. I'm not a web developer though, so I really don't know if the state of the art is actually being used there.

        A snag though is if a CRL goes out invalidating a cert, then that cert is presumably no good anymore. At what point is the certificate no good in the web? Do 2 out of 3 votes of "it's good" make it work, or does a single "don't trust it!" suffice to in

      • There is nothing stopping you getting two certs and presenting both. TLS has supported this for a long time.
        • This may be not doable, depending on the HSM. Storage on those devices can be limited, and storing multiple private keys may not be doable. Of course, this is definitely an option, but it isn't really a mainstream one.

          • by unrtst ( 777550 )

            IMO, the reason getting multiple certs isn't common is the same sort of reason as to why most do not have a mirror of their entire infrastructure mirrored to an offsite location.

            For example, in the old datacenter way, getting space in at least two data centers with duplicate hardware in both, both fully configured and ready to run, and able to cut over to the cold spare with minimal effort. In the cloud, duplicating your whole setup at a region facility. It's too much overhead financially and in man hours f

  • Why would the CABF, whatever that is, be very strict and require revocation in less than 24 hours? There's no middle ground? Suspension maybe? Hell, when I don't pay my hosting bills, the website is parked for a while until I pay and the DNS resolves again, but the site isn't "revoked" or removed.

    • by Xenx ( 2211586 )
      The revocation isn't functionally too dissimilar, in terms of overall effect, to suspending/parking. The site/service is still there, it just won't be considered secure until it's fixed. As for why you would do it within 24hrs, improper validation is a huge potential security risk. While we know roughly how/why things went wrong, so the risk here is minimal, you risk the whole process losing credibility when you start making exceptions.
    • by madbrain ( 11432 ) on Monday July 29, 2024 @10:52PM (#64665732) Homepage Journal

      You can revoke with a reason code of certificateHold. The behavior is identical as for other reason codes, except you can unrevoke the certificate later if the revocation was in error. It doesn't appear that it applies in this case.

    • by rta ( 559125 )

      It makes sense in the context of an actual fraudulent or potentially fraudulent cert. i.e. it puts the onus on them to not leave questionable certs sitting out there giving clients a false sense of security.

      That said, in THIS case it appears to be silly and unnecessary, of the type "well the contract was signed in blue ink and our rules say black" sort of way.

      But compliance / security field, as practiced, isn't generally known for balanced evaluations of risk... it's mostly legalistic rule following.

    • Why would the CABF, whatever that is, be very strict and require revocation in less than 24 hours?

      To minimise fraud you need to quickly address the mechanism using it. In this case it happens to be innocent, but the reality is the rules probably don't apply different times to different revocation reasons.

    • There's no way to suspend. The suspend functionality is a temporary revocation (and unsuspending is removing the revocation from CRLs and OCSP-responders). Revocation is inherently broken on the web. No browser checks OCSP or CRLs. OCSP-stapling is optional and the Web-PKI is in the process of obsoleting OCSP completely. (this is the reason for the drive to ever shorter validity periods: as there's no way to revoke a certificate in any meaningful way, you need short certificate lifetimes to get bad certif
      • by leonbev ( 111395 )

        Unless your client or vendor happens to use a DigiCert certificate that's impacted. Then, they're screwed. For those clients, this is going to be like CrowdStrike all over again.

    • This all reads like autistic screeching of the "it says right here in the rules!!!" type.

      As if there is a security problem.
      As if CABF will revoke DigiCert's authority for taking care of their customers.
      As if nobody takes a 2-week vacation in the summer.
      As if SMB's don't just have one guy who knows how to deal with certs.
      As if they aren't knowingly going to cause (at least a few) outages
      As if they aren't going to get their asses sued.

      Digicert is killing its reputation for customer service.
      Digicert is going t

      • Apparently the CABF never realized how its rigidity cheapens the whole concept of this mechanism for fraud prevention. Using revocations as a form of digital Wite-Out sends a different message altogether. I used to think cert revocations were reserved for only serious, security related issues. In the past I would probably trust a signed cert that had simply expired recently, but no way would I trust a revoked cert! Well the joke's on me, now I know better.

        I guess the message is to go ahead and accept wh

        • by dgatwood ( 11270 )

          Apparently the CABF never realized how its rigidity cheapens the whole concept of this mechanism for fraud prevention. Using revocations as a form of digital Wite-Out sends a different message altogether. I used to think cert revocations were reserved for only serious, security related issues. In the past I would probably trust a signed cert that had simply expired recently, but no way would I trust a revoked cert! Well the joke's on me, now I know better.

          I guess the message is to go ahead and accept whatever cert is presented by whatever server. There's no way of knowing the significance of a revocation when they can be revoked for any trivial reason their perseverating hiveminds can conjure up. It could be anything from the validity period having one extra second, to missing an obscure field (cPSuri) that nobody ever looks at, or even knows of its existence.

          Since it's almost always going to be a corrective type revocation, who actually cares? I'm hearing it lound and clear. CABF says just click Continue and go about your day.

          If this happens more than a tiny number of times on more than a tiny number of websites, yeah, that's exactly what people are going to start doing.

          But IMO, the real problem was the "HTTPS everywhere" nonsense. A website that is purely informational and doesn't take logins doesn't really need HTTPS. Nobody is going to bother pretending to be a website to give fake information about something. So except when the content of the website is potentially sensitive (i.e. sites that you don't want other people to

    • by ebunga ( 95613 )

      See, the CA/Browser Forum is run by Google, and maybe the other Browser out there too, and the CAs, which are beholden to Google because Google Knows Best. They have no concept of the real world. This is why they have an argument about what "within 24 hours" means, and why they push everyone towards fragile automation that ultimately reduces security rather than improves it.

      That being said, if I had an improperly issued certificate on one of my servers, even one I wanted and requested, I would expect it to

  • "affecting about 0.4% of their domain validations"
    Ok

    "This could literally be thousands of SSL certs."
    Wake me when it's 8.5 million.

    "This could take a lot of time and potentially cause outages worldwide starting July 30 at 19:30 UTC"
    ...or not because customers have been notified.

    "Be prepared for a long night of cert renewals. DigiCert support line is completely jammed."
    Why do you need to call them to renew a cert? Just use the self-service web interface.

    I'm expecting a total nothingburger fr
    • by Xenx ( 2211586 )

      Why do you need to call them to renew a cert? Just use the self-service web interface.

      If you've ever worked in support, you know that there will always be people needing to call. I've spoke to many smart people, that are complete idiots under the wrong circumstances.

      • Maybe 'those' people shouldn't be managing domain certs in the first place XD

        This might cause an issue for a few small sites, but I'll be surprised if anyone besides their own close circles notice.
        • by cstacy ( 534252 )

          Maybe 'those' people shouldn't be managing domain certs in the first place XD

          Maybe the UI on the self-serve web portals shouldn't be complete unusable shit. (Not maligning DigiCert here -- I've never seen theirs. Well, I probably have used it actually just not lately and I don't remember at all. Just a general comment on those "stupid users".)

          I've been programming since 1973, in all the most advanced environments, and the first thing I tell people about computers is: "They tell you this is user friendly. It is not. It is crap. It's not your fault you think it's hard to use. It is.

          • by ctilsie242 ( 4841247 ) on Tuesday July 30, 2024 @12:14AM (#64665810)

            Renewing Web certs can be an exercise in frustration with some providers. How hard is it to kick up a .csr, get a .crt back?

            Where the real pain point comes in are various apps. Each of them want the certificate file with the private key in a completely different format. Some want the .key file as a separate item. Others want it catted to the .crt file. Still others want the entire cert chain from root to intermediaries to the end cert, with the private key dangling along for giggles. Still others, you just say hell with it, slap an nginx or Apache proxy on front of it, and call the job done, because restarting a proxy after a cert renewal is a lot easier than hoping edits to the their weird key storage thing actually work.

            Also, do the relatively short lives of 6-12 months for non-LE certs actually increase security? I'm not seeing this.

            How about a compromise on this. If the certificate is generated in a HSM, and the cert has an attestation certificate showing it was generated in a HSM, then allow it to have up to a ten year lifetime. This way, if hardware security is involved, and this could be even separated from low level HSMs to ones that have a ton more security and anti-tamper measures, then the worry about the private key material escaping to the wild is not an issue.

        • by pjt33 ( 739471 ) on Tuesday July 30, 2024 @02:15AM (#64665980)

          Some companies might have so many certificates that they need a dedicated person to manage them, but plenty of companies will have one certificate which needs renewing once per year, and that's not frequent enough to become second nature. On top of that, the timing is such that some companies are undoubtedly finding that the one person who's done it before is on holiday and some other muggins needs to figure out how to do it under extreme time pressure. I would expect that at least some of the calls are people checking "Do I really need to do this? Are you sure you can't fix things without me doing anything?"

    • by cstacy ( 534252 )

      "affecting about 0.4% of their domain validations" "This could literally be thousands of SSL certs."

      Wake me when it's 8.5 million.

      These are not client certs. they are servers. So if "thousands" means 8,500 sites with just 1,000 users each, there ya go.

      But a lot more people than 8.5 million were affected by CrowdStrike. Several orders of magnitude more, I suspect. A lot of the affected machines were not servers, but things like management or customer service rep PCs. Even if not every PC someplace had CrowsStrike on it, just some, there are major ramifications because it's exactly the ones they thought were sensitive enough to enroll.

      K

    • by leonbev ( 111395 )

      Have you ever had the misfortune of renewing a certificate from DigiCert? For some odd reason, they randomly flag cert renewals and require the requestor to provide proof that they own the business that they're representing. Which, if you're an entry level IT person with no easy access to accounting or the C suite, is pretty damn hard to do.

    • by leonbev ( 111395 )

      It just needs to take down a few big web sites that serve a few million users, and this could be another big outage.

      I'd imagine that a lot of big companies who use DigiCert certificates are bureaucratic monsters who can't even approve a password change in 24 hours. They'll get hit by this hard.

  • My organization is probably going to get hit by this for a few of the certificates. I'm away right now so I can cross-check their entire list of certificates to see which ones are going to be revoked, but I have a feeling it's going to be a few .

    Unfortunately, my organization keeps promoting and assigning responsibilities to people who are loyal to management and tow the company line instead of the people that are technically and security conscious So we end up with people who have no background in x.509 or

    • I apologize for the terrible grammar on the post above. I was trying to be as eloquent as possible enunciating every word correctly and slowly to Google while I was dictating that but it seems like it has failed completely trying to understand my speech to transcribe it into text.

    • by micheas ( 231635 )
      An honest question: Why not move most of the certificates to something that can be renewed with certbot or one of its clones?
      • by ebunga ( 95613 )

        I really love the fact that whole point of short certificate lifetimes is "because bad guys might compromise a system and be there for a long time with a valid certificate" while certbot and the ACME process allows for a compromised machine that never had a certificate to be automatically issued a certificate without any human oversight. But sure, once the compromise is discovered three years later the certicate will automatically die in 90 days, so I guess that's a good thing?

        • by dgatwood ( 11270 )

          I really love the fact that whole point of short certificate lifetimes is "because bad guys might compromise a system and be there for a long time with a valid certificate" while certbot and the ACME process allows for a compromised machine that never had a certificate to be automatically issued a certificate without any human oversight. But sure, once the compromise is discovered three years later the certicate will automatically die in 90 days, so I guess that's a good thing?

          The whole model is garbage. And heaven help you if you start running into validation issues because one of your secondary domain name servers doesn't respond. The LetsEncrypt name servers apparently cache failures to respond, and don't retry with the other name servers on the list, so if anything is even slightly wrong (for example, because of a secondary being down), you'll spend hours trying to get it to work, waiting for the DNS changes in the domain record to propagate to all of the caching root-adjac

    • A lot of talent gets wasted in America.
  • I got off lightly. Only 3 of mine needed reissuing. But 24 hours notice is insane. We got the email After hours yesterday so by the time i read it i had about 9 hours left. I feel for anyone that might be on holiday or not checking emails.
  • So DigiCert, in the process of having a certificate requestor validate their control of the domain a certificate was requested for told them to add the CNAME . to their DNS to prove they had control. The requestor did this. DigiCert verified that the CNAME was added. DigiCert issued a certificate for the requested domain. Now DigiCert is saying that because they did not tell the client to add _., they need to revoke the certificate and try again.

    Am I understanding this correctly?

    Sure, fix the poli
    • Forgot, can't use angle braces ... client added [random].[requested-domain] as requested when the request should have been for _[random].[requested-domain]
      • by dgatwood ( 11270 )

        To the best of my understanding, yes, that is correct. And this is utterly the most idiotic reason for screwing over site admins with a 24-hour panic attack that I've ever heard in my life. If anybody still uses DigiCert after this week, it will be a miracle.

Duct tape is like the force. It has a light side, and a dark side, and it holds the universe together ... -- Carl Zwanzig

Working...