CircleCI Says Hackers Stole Encryption Keys and Customers' Secrets (techcrunch.com) 23
Last month, CircleCI urged users to rotate their secrets following a breach of the company's systems. The company confirmed in a blog post on Friday that some customers' data was stolen in the breach. While the customer data was encrypted, cybercriminals obtained the encryption keys able to decrypt the data. TechCrunch reports: The company said in a detailed blog post on Friday that it identified the intruder's initial point of access as an employee's laptop that was compromised with malware, allowing the theft of session tokens used to keep the employee logged in to certain applications, even though their access was protected with two-factor authentication. The company took the blame for the compromise, calling it a "systems failure," adding that its antivirus software failed to detect the token-stealing malware on the employee's laptop. Session tokens allow a user to stay logged in without having to keep re-entering their password or re-authorizing using two-factor authentication each time. But a stolen session token allows an intruder to gain the same access as the account holder without needing their password or two-factor code. As such, it can be difficult to differentiate between a session token of the account owner, or a hacker who stole the token.
CircleCi said the theft of the session token allowed the cybercriminals to impersonate the employee and gain access to some of the company's production systems, which store customer data. "Because the targeted employee had privileges to generate production access tokens as part of the employee's regular duties, the unauthorized third party was able to access and exfiltrate data from a subset of databases and stores, including customer environment variables, tokens, and keys," said Rob Zuber, the company's chief technology officer. Zuber said the intruders had access from December 16 through January 4.
Zuber said that while customer data was encrypted, the cybercriminals also obtained the encryption keys able to decrypt customer data. "We encourage customers who have yet to take action to do so in order to prevent unauthorized access to third-party systems and stores," Zuber added. Several customers have already informed CircleCi of unauthorized access to their systems, Zuber said. Zuber said that CircleCi employees who retain access to production systems "have added additional step-up authentication steps and controls," which should prevent a repeat-incident, likely by way of using hardware security keys.
CircleCi said the theft of the session token allowed the cybercriminals to impersonate the employee and gain access to some of the company's production systems, which store customer data. "Because the targeted employee had privileges to generate production access tokens as part of the employee's regular duties, the unauthorized third party was able to access and exfiltrate data from a subset of databases and stores, including customer environment variables, tokens, and keys," said Rob Zuber, the company's chief technology officer. Zuber said the intruders had access from December 16 through January 4.
Zuber said that while customer data was encrypted, the cybercriminals also obtained the encryption keys able to decrypt customer data. "We encourage customers who have yet to take action to do so in order to prevent unauthorized access to third-party systems and stores," Zuber added. Several customers have already informed CircleCi of unauthorized access to their systems, Zuber said. Zuber said that CircleCi employees who retain access to production systems "have added additional step-up authentication steps and controls," which should prevent a repeat-incident, likely by way of using hardware security keys.
Re: (Score:3)
One-Click Bankruptcy (Score:2)
Rightful ownership of session token (Score:1)
Guess they didn't follow best practices! (Score:2)
Hardware security modules, anyone? (Score:2)
What ever happened to just using HSMs for anything valuable? The purpose of a HSM is that even if a place is completely compromised by attackers, the worst they can do is sign/decrypt stuff.
If a key is of -any- value, it needs to go into a HSM... or even better, be generated in the HSM, and never leave the appliance, other than for encrypted backups or syncing to another HSM. Valuable SSH keys? NitroKey or YubiKey HSM is one example. GPG signing keys or RPM signing keys? Those should never have their p
Re: (Score:2)
Unless you're running all your traffic through the HSM then the key winds up in memory where it can be stolen.
Partitioning/subkeys/revocation schedules - at some point nothing is foolproof when you have live systems on the Internet.
At least with PC hardware; so far.
Re: (Score:2)
For things like executable or GPG signing keys, the amount of signatures being done is relatively minimal. Similar with root keys. Intermediary keys, it may make more sense to have that on a secure desktop, just for more signings per second. Something like Datadog's RPM signing key [techdigipro.com] is very concerning. Ideally, that key should have been in a HSM, so it could have never been exposed.
Re: (Score:3)
Well, unless your a CI system, in which case your doing it all the time.
CircleCI is a CI system.
The only ones to blame here (Score:4, Insightful)
Re: (Score:2)
What's the alternative though? A well paid IT department that hires skilled people to keep your stuff safe? DevOps?
From a business perspective, occasional problems like this are probably not going to cost them much more than some time to regenerate keys. Most of what was stolen probably has little value to anyone else.
Re: (Score:2)
I use Jenkins on dedicated machines inside my company's own network.
Another option may be to run GitLab, Jenkins or some other CI infrastructure in the "cloud", but with a more reliable vendor than CircleCI (for example AWS, Azure etc) and on a closed down network with no public access (for example by enforcing VPN).
Giving your secrets to any company who is willing to take your money is just incredibly foolish. Unfortunately, it's what lots of company's do today.
If bad actors gets access to your build syst
Re: (Score:2)
Encrypt using your own keys before you upload anything to the cloud. Letting your cloud provider encrypt for you using their keys is just dumb.
Re: (Score:2)
For a CI the keys have to be on the server, otherwise how is it going to build the software?
Your suggestion makes sense for file storage, perhaps that's what you were thinking of.
Re: (Score:2)
Re: (Score:2)
Here is one way to do it - IBM Secure Execution for Linux: https://www.ibm.com/docs/en/li... [ibm.com]
Re: (Score:2)
The key is 'insecure computer', not 'cloud'. There is a difference. You can have a secure computer as part of the cloud, and that secure computer will not let the 'someone else' access your secrets.