German Health Insurance Card CA Loses Secret Key 174
Christiane writes "The SSL Root CA responsible for issuing the German digital health insurance card lost its secret private key during a test enrollment. After their Hardware Security Module (HSM) dutifully deleted its crypto keys during a power outage, it was all 'Oops, why is there no backup?' All issued cards must be replaced: 'Gematik spokesman Daniel Poeschkens poured scorn on the statement that Gematik had insisted on the service provider carrying out a test without backing up the root CA private keys. "We did not decide against a back-up service. The fact of the matter is that the service provider took over the running of the test system, so it also has to warrant its continuous operation. How it fulfills this obligation is its own responsibility."'"
Oh c'mon, be fair! (Score:5, Funny)
Not even a month ago you chided them because there were too many copies (some of them even offsite, they just didn't know who had them now), now you chew them out for having too few. Make up your effing mind!
Re:Oh c'mon, be fair! (Score:4, Funny)
This kind of organisation usually has a backup somewhere, they just have to find it. Its usually backed up on a post-it note somewhere. Maybe they should ask all of us to look for it, on the sides of our monitors.
Re:Oh c'mon, be fair! (Score:5, Funny)
My Day 1:
I actually found the administrator password on a post-it note on the back of the server's CRT monitor while cleaning the server room.
"Fucking amazing" I said out loud, and as I pulled it off, on the back was the AmEx credit card number, expiration date, and 3digit pin for our organization to order IT stuff.
Then I noticed on the left underside of the CRT there was another post-it that said Ctrl Shift Alt Num+....so I pressed that and up came a hidden menu of hidden apps running (SysTrayX + a sketchy prog to hide services in TaskManager), 90% of them illegal. Also uTorrent was running, seeding about 50 anime series buried deep within the network and using about half of the T3 connection's throughput.
And to top it all off, I deduced that the server had never had a fresh install of Windows. It used to have NT Server, then they used software to upgrade it to 2000 Server, and software again to upgrade it to Server 2003. ......
Day 7:
I get a call from the old IT guy asking me whats wrong with the connection, and I told him I reinstalled Server 2003, deleted his anime cache, changed the WPA-PSK keys from 1111111111 to something way more secure, reported the AmEx card as stolen to get a new one, changed the admin password and set password age limits on all accounts, and replaced the rootkit infected SCSI drives with new ones that would last longer. Also, I told the managers that his 5000$ quote for network-wide unlimited antivirus was utter bullshit and that he only got a cracked key for Norton 2003 and installed it only on the server, and prolly pocketed the money.
Damn dude was like "BUT I DIDNT BACK UP THE ANIME TO DVD YET!!!". Now I love anime as much as the next person, but I think he has other stuff to worry about at this point.
But you know what got me the most mad and prompted all of this? The server was named Odie, and the computers were all garfield characters.
CALVIN AND HOBBES FTW!!!!
Re:Oh c'mon, be fair! (Score:4, Funny)
Oh, and his DAT72 backups had been failing for the last 2 years and he had never checked the logs.
Good thing he left to start his own business! /shudder
Re: (Score:3, Funny)
Day 8:
You got fired, and the system got "restored" because your "fixes" halted the whole "business".
It was a sad day.
Re:Oh c'mon, be fair! (Score:5, Funny)
Heh...I'm actually just doing a paid internship at a non-profit after their full-time guy left. It was supposed to end on May 1st, but hey I guess they love what I've done.
Got them a cheap dedicated backup system, updated all the systems and reinstalled an NLite-ed XP on every computer, and moved them from Exchange to Google. Oh, and the lab computers run Ubuntu.
They also loved it when I found the IT guy's secret paypal business account with 3000$ sitting in it that was supposed to be used for something else (battery backup replacement batteries). Putting passwords in a file on the administrator's desktop called "passwords for everything.txt" is sooooo helpful for when you're trying to be sneaky.
Seriously, this shit is a soap opera of IT-isms.
Re: (Score:2)
Are you making this up? If not, your company should take criminal action against the prior employee.
Re: (Score:2)
Done months ago. And no, I don't make shit up. They settled (out of pity) for the money they found that he took after going through their books, but he's pretty much blacklisted in the city we're located in so he moved.
The whole branch nonprofit is over to free software (minus xp/2003 on the dell computers they have and office 2003/outlook 2007 VLKs bought from techsoup).
The last thing they need to do is switch to voip and get rid of the horrible lease they're on for their ancient PBX that this idiot convin
Re: (Score:2)
Re: (Score:2)
I don't have anything agenst Partners, you were locked in harder than Microsoft with those systems.
Re: (Score:2)
Re: (Score:2)
Since you don't have your e-mail listed, I'm posting here.
I'm a highly underpaid IT admin working at a 501(c)(3) trying to admin a woefully underequipped, underpowered, and understaffed network. I'm having a hard time figuring everything out. If I could toss a few questions your way and you'd have the time to answer them, would you please e-mail me?
ihmhi6@gmail.com
Re: (Score:2)
If everything fails, keep browsing through various pages trading in that stuff, you'll eventually find it...
Re: (Score:3, Insightful)
Its usually backed up on a post-it note somewhere.
For a root CA private key, better be a big post-it note
(or written in really small letters)
Re: (Score:2)
Common keysize is 2048 bits which makes 256 8-bit words or characters.
Re: (Score:2)
that's 512 0-9A-F (or 65536 0-1 I suppose) to write down. And I thought trying to enter a WEP key in windows was bad...
Re: (Score:2)
And please don't forget to flush.
Re: (Score:2)
They can have my copy, I hacked their servers last week ;)
Wrong Title, Wrong summary (Score:4, Informative)
Once again, misleading title to a different summary.
For fuck's sake, the Germans didn't lose the key.
The SSL Root CA lost that.
Get the facts right.
For a second i was wondering how Germans could that stupid. That is unlike the Germany i know. And exactly as i suspected, the German insurer had been insisting the root CA for backup while the CA thought it was unnecessary.
Is it the German company's fault?
Re:Wrong Title, Wrong summary (Score:5, Funny)
After all, we all know Germans are exact and punctual, Poles are thieves, Russians are drunk and Fins are even more so. Oh, and Mexicans are lazy and US people are simple minded. Any stereotype missing?
Re:Wrong Title, Wrong summary (Score:5, Informative)
Any stereotype missing?
yes.
we British are all of the above.
Re:Wrong Title, Wrong summary (Score:5, Funny)
Not only that, they have really weird tastes, too. In food and bed. Sometimes at the same time.
Re: (Score:2)
And how exactly do you pull off being exact and punctual while being sloppy and unable to figure out what time it is from being drunk?
Re: (Score:2)
And how exactly do you pull off being exact and punctual while being sloppy and unable to figure out what time it is from being drunk?
Practice... lots and lots of practices. Speaking of which it's time for me to do some practicing.
Re: (Score:2)
Speaking of which it's time for me to do some practicing.
That should be "practising" - oh, and we're good at pedantry too.
Re: (Score:2)
how exactly do you pull off being exact and punctual...
They tattoo the pub opening time and location on their arms.
Re: (Score:2)
Well "exact" could mean "accurate" or it could mean "precise".
I take it to mean it to mean that Brits in their inebriated state have a gift for putting their fingers and other assorted appendages in precisely the wrong place.
Re: (Score:2)
Free +5 funny for whoever follows up with a comment about US beer
like oh, "americans like their beer cold?
Re: (Score:2)
Who wants beer at freakin' room temperature?? Ick!
But, I do like the Monty Python quip:
"We find your American beer is a little like making love in a canoe..."
"Making love in a canoe?"
"Yeah, it is fucking close to water..."
Re: (Score:2)
So, you're punctual, thieving, drunk, lazy, simple minded, and British.
So, just like the Irish then, you mean?
Re: (Score:2)
So, you're punctual, thieving, drunk, lazy, simple minded, and British.
So, just like the Irish then, you mean?
The Irish don't like being called "punctual", you insensitive clod!
(And they really hate being called "British")
Re: (Score:2)
Any stereotype missing?
yes.
we British are all of the above.
What about the Irish?
Re: (Score:2)
I must take issue with your sweeping nationalistic statement. Poles aren't theives - that's Romanians. Poles are honest. Crap at plumbing, but honest.
Re: (Score:2)
The "crap at plumbing" is due to a little known fact. Namely that Poland invested zero into the infrastructure in the western areas they got after WW2, fearing they'll eventually return it to Germany. Until not too long you could find pipes dating back to pre-1940.
Re: (Score:2)
Yeah, but the Plumbers that did those pipes weren't Polish...
Re: (Score:2)
.... and US people are simple minded. Any stereotype missing?
Simple minded? I thought we were just fat and lazy.
Re: (Score:2)
You're fat and lazy because you're simple minded. And if you weren't so simple minded, you'd be able to figure that out.
Me, I'm Canadian with British parents. So apparently I'm exact and punctual while stealing booze, but I apologize politely to the shopkeeper for swiping it.....
Re: (Score:3, Insightful)
Well, we DO know that they are awfully good at writing numbers down. Sometimes even up the arm.
Re: (Score:2)
Re: (Score:2)
That all Oriental people are great drivers, and the men are well endowed?
Re: (Score:2)
Re: (Score:2)
I thought Mr Patel was the guy who has kept very quiet his monopoly on every corner shop everywhere.
Re: (Score:2)
the only 7-11 near where I live is run by hippies
Just a wild guess, but are you from Portland by any chance?
Re: (Score:2)
Question then becomes, can you still call it a stereotype if it's true?
I think the definition of stereotype is an exaggerated generalization, meaning that while it may be true, stating that it's true to a greater degree than factual is where it crosses the line.
"ALL the 7-11's in Seattle are staffed by Indians after 9pm" - stereotype
"80% of the 7-11's in Seattle are staffed by Indians after 9pm" - possibly fact.
Re: (Score:2)
Re: (Score:3, Insightful)
The summary even states that Gematik insisted on a back-up less operation, and then provides a quote explicitly stating that they did no such thing! Slashdot: doing for editorial accuracy what Fox does for editorial neutrality.
Re: (Score:3, Informative)
Gematik commissioned D-Trust to provide the root CA as a service. D-Trust managing director Matthias Merx stated "Gematik decided to 'do without a back-up'. As a service provider, we have to accept that."
From the article and summary:
"Gematik spokesman Daniel Poeschkens poured scorn on the statement that Gematik had insisted on the service provider carrying out
Re: (Score:2)
I'm guessing that our self-evidently poor (well, my self-evidently poor) reading ability is to blame somewhere.
Re:Wrong Title, Wrong summary (Score:4, Interesting)
"We did not decide against a back-up service ..."
That double negative sounds awful like "At the time, we didn't know what they were asking":P I guess its just with personal experence. Evey time I hear a manager use double negatives to defend a decision, its because they didn't really know what they were deciding in the first place. Atleast in IT.
Re: (Score:3, Informative)
The title/summary are not necessarilly incorrect, just ambiguous. English can do that, and if you aren't paying attention your meaning can be taken in a way other than you intended.
In this case, there are a few ways to read "German Health Insurance Card CA":
1.) The Health Insurance Card CA of German origin
2.) The CA for the German Health Insurance Card
3.) The Card CA for German Health Insurance
4.) The Insurance Card CA for German Health
Obviously they aren't saying 3 or 4, those work gramatically but don't
Re:Wrong Title, Wrong summary (Score:5, Insightful)
This is why managers (especially the MBA types) love outsourcing of everything. It is also in part because numbers and KPIs are so much more easy to manage than actual people. But mainly, by outsourcing a function you also get to outsource the responsibility for that particular function. If things go tits up, the worst you'll be blamed for is picking the wrong service provider, or perhaps not monitoring a particular KPI properly. Minor stuff.
I've seen plenty of managers like that, and I have heard a variation of that one line all too often.
Re: (Score:2)
Hah. Why do you think the consultants were thick as flies in government during New Labour's reign?
Yup, the Shaggie defence (it wasn't me)..
Re: (Score:2)
NSA/CIA (Score:4, Funny)
Maybe they should check with the NSA or CIA? They've got a backdoor into EVERY system, and may still have the key saved on a laptop lying around somewhere.
Re: (Score:2)
Could be worse (Score:5, Insightful)
Re:Could be worse (Score:4, Interesting)
What's worst about it is that this is probably presumed to be worse. Had the key be stolen, they'd probably not even report it because business could continue as usual, maybe nobody finds out...
Re: (Score:2)
Mod parent up. In the serious crypto world, this is a good thing, provided it doesn't happen too often. Sometimes you're going to lose a key, because, for security reasons, you don't keep extra copies. You have a procedure for issuing new keys when this happens, which you're routinely doing anyway.
Re: (Score:2, Interesting)
It could be worse, but this incident exposes a design flaw: The loss of a private key should not stop them from issuing new cards which are compatible with the existing cards.
If a CA key is lost, then there should be a layer above it which can create a new CA key. Cards are checked against the top CA public key, so the old and the new cards can both be verified. Because the top CA is only used to create intermediate CAs, its private key can be kept safer than the key of a CA which is regularly used for sign
This would never happen in Britain (Score:3, Funny)
It would easily be found be searching the nearest pub car park for USB keys, or checking the train that the relevant civil servant travelled home on.
Public Key Infrastructure (Score:2)
Re: (Score:2, Insightful)
That's just silly. They obviously take security seriously enough that they found re-issuing all of their certs preferable to adding a second storage place for the private key, thus doubling the possibility of the system being compromised.
If the key had been compromised, that would be a breach of trust. This is more an example of the fact that as security increases, usability decreases.
Re: (Score:2)
Re: (Score:2)
Why do they have to issue new keys? I'd think that as long as their public key is still known, that all the issued signed keys would still be valid. They'd just have to use a new key pair for any new signed documents.
Re: (Score:2)
Damn good question. Losing a signing key doesn't mean the signatures can't still be checked.
Re: (Score:3, Insightful)
Actually, I can think of a reason, after all. Since this CA no longer has the ability to revoke prior signatures made with that key, then that key can no longer be trusted as a signer. You can check to see if a CA has certified something, but there's no way to check to see if the CA changed their mind, because the CA no longer has a way to say that.
Re: (Score:2)
All the issued signed keys are still valid. That's the problem.
There's no way to now invalidate a signed key that was issued with this CA.
Normally you can revoke a certificate, so it's no longer accepted. OpenVPN, for example, allows this, because you can set it to check a Certificate Revocation List. Any employee gets fired, certificate gets compromised, or whatever, and you add it to the CRL, and that client can no longer connect.
Right now, if they find that a health card was issued fraudulently, or fo
Re: (Score:2)
Actually, people only give you half the story. Not only can existing keys not be revoked (serves you right for not setting a timeout), you can also not create new keys.
The second part is where it gets entertaining. In order to cure that problem (which is something you want to sort if you're about to waste a lot of money on issuing certs) you will have to generate a NEW root certificate. However, there can only ever be one root, putting a new root key in the system means you have just broken the chain of
Re: (Score:2)
Re: (Score:3, Insightful)
PGP Desktop has this option. You can share a key and split it among people, where x amount of y pieces are needed to recover the original key, where both x and y are user selectable values.
However, if a key is a top root CA key, you would not be using it on a general purpose computer. You would have the key generated in a HSM and stored there, where someone can perhaps use the key to sign and decrypt stuff, but would have to go to a lot of trouble to get past all the hardware tamper evident stuff in the H
You can fall off the road on either side (Score:4, Interesting)
Mistakes happen, of course, and certificate infrastructures can be enormously complex. But if you're going to do any kind of risk mitigation, the absolutely most basic place to start would be with these two scenarios.
Re: (Score:2)
certificate infrastructures can be enormously complex.
This is the problem: simplicity is the key to security. A complex system is just one with more places to hide exploits.
Re: (Score:2)
So, it's kind of like the optimist/pessimist thing, right?
As an optimist, I'd say that least they didn't fail in the worst possible way.
The pessimist in me thinks I should get a bit more than "not failing in the worst possible way" when I pay somebody a barrel of cash to hash a couple numbers for me.
Re: (Score:3, Informative)
As an optimist, I'd say that least they didn't fail in the worst possible way.
The pessimist in me thinks I should get a bit more than "not failing in the worst possible way" when I pay somebody a barrel of cash to hash a couple numbers for me.
No, that's also the optimist in you.
Cheers. :)
Re: (Score:2)
Re: (Score:2)
Chain the cards to truck wheels, like the bathroom key at the service station.
Re: (Score:2)
I wouldn't have thought keeping the root online at all times was particularly sensible. At least I seem to remember that this was some of the point of the hierarchical certificate system. You generate the root cert, then some tier 1 certs and turn off the root, put it in a cupboard* use the tier 1 certs to generate more, this way you don't compromise the absolute top of the hierarchy. Which should make the fail moments slightly easier to manage.
*preferably one with a good lock and lined with steel.
Re: (Score:2)
I often speak in favor of operating a certificate hierarchy, as you've described. But notice that spreading the risk across multiple points of failure not only increases the intrinsic risk of failure, it multiplies the cost of managing the certificate infrastructure. A secondary risk is
Rootkeylosin! (Score:3, Funny)
Q: How do you learn every German swear word in about 20 seconds?
A: Tell the German admin that you lost the root key.
Re: (Score:3, Insightful)
I'm confused (Score:5, Insightful)
I'm confused, isn't this sort of problem exactly why you carry out system tests?
Sending out new cards to card testers during a systems test is hardly extraordinary.
Re: (Score:2)
Re:I'm confused (Score:4, Informative)
Matthias Merx, the firm's managing director, told heise online that following a voltage drop, something happened in D-Trust's "Trustcenter" that does occasionally occur. "The HSM independently deleted the data because it suspected an attack."
Translation? "Someone unplugged the backup power supply before setting the proper mode in the card because we didn't fully understand how sensitive the card is for root CA certs"
Merx explained that "Gematik decided to 'do without a back-up'. As a service provider, we have to accept that,"
Translation? "We asked Gematik that it might be a good idea to back it up and they said its fine its just for testing." or "We recommended to Gematik to back up the card before shipping it to us. They shipped it to us and we just shrugged our shoulders." Bonus points if you guessed they asked a low level manager at Gematik who thinks CA is the first two letters of a cat.
Gematik spokesman Daniel Poeschkens poured scorn on the statement that Gematik had insisted on the service provider carrying out a test without backing up the root CA private keys. "We did not decide against a back-up service. The fact of the matter is that the service provider took over the running of the test system, so it also has to warrant its continuous operation. How it fulfils this obligation is its own responsibility."
Traslation? "Gematik is taking NO RESPONSABLITY WHATSOEVER for doing any safty checks before giving our root ca to an outside vendor."
All in all its not a big deal though. It looks like they just lost the issuing CA and not the revoke keys. It looks like they can still authenticate too. Now if this was the MAIN system germany with 80+ million plus medical cards? I think people are going to be shot:P
A drop in voltage? (Score:2)
Place blame (Score:5, Funny)
What is "CA"? (Score:5, Funny)
The problem here is that the company deleted the certificate-printing program since they thought someone was trying to hack in and print more certificates for themselves- no one is THAT special so they had to stop him. They forgot to have another program ready to print more certificates, so now Germany is under threat of entering a depression since they no longer get certificates telling them how special they are.
On a serious note: I don't follow this article very well with all the acronyms being spelled out but not explained, and no background knowledge of anything going on here. If someone would care to explain what is going on here to someone that has never heard the term CA, you should get a +5 informative easily.
Re:What is "CA"? (Score:5, Informative)
I will simplify, but basically a CA (Certificate Authority, that much of the parent wasnt a joke) is a server that creates encryption certificates. In this case, SSL certificates. For example, when you goto https://mail.google.com/ [google.com] that SSL certificate was created by the Thawte SGC CA. Thawte is one of many companies that you can pay to create you an SSL cert, so your users can communicate with your server via https.
The CA itself also has an encryption key, which is stored on hardware. In some cases its a PCIe board, others its a removable PCMCIA card, etc. This particular CA used an add-on board which lost power during operation, wiping out its only key. The board seems to have been working as intended, preventing attack (removal of board, which would cause power loss) by wiping itself.
Without that key, the CA can no longer create revocation lists (CRLs, lists of certs a CA has created that have since been revoked or expired) or any new certs. They are dead in the water, also causing every cert they have ever made to become invalid as they can no longer be checked against a recent CRL. They have to start from scratch, recreating every_single_cert.
This was only a test system, but if this happened in reality 80 million Germans would have invalid health cards. At least they discovered the value of a backup during testing.
Re: (Score:2)
BTW: The private key doesn't have to be stored on a card, it can be just as easily set up as a file on a disk.
I looked into becoming a CA once in order to support a state contract - we were just going to use OpenSSL and a strongly physically secured computer with no network access.
Re: (Score:2)
I looked into becoming a CA once in order to support a state contract - we were just going to use OpenSSL and a strongly physically secured computer with no network access.
You might use a two-layer system. Have a master CA and a production CA. The master CA is kept offline (and probably normally unpowered and in a locked fire-proof safe) and is only used to sign the production CA. The production CA issues all the certs that you're actually using to secure normal servers and users. (Actually, you can go with more than one layer of production CAs, but that's less important.) The idea is that the master only needs to be used extremely rarely, so it can be secured using extra str
Re: (Score:2, Informative)
A few points first:
- A CA doesn't only create encryption certificate. It can create a variety of certificates, including Windows Logon, Signing certificates, etc. It all depends on the Certificate Policies that are configured on the CA.
- We have no information the CA was indeed issui
Best practices (Score:2, Informative)
Best practices about CA management says you should have your secret key in a (physical) safe. Better yet, divide it in two pieces and put it along the passphrase in three different safes (part1+pass,part2+pass,part1+part2), so you can't lose key access even if you lose one safe, and nobody can take the key by opening a single safe.
Re: (Score:2)
And where do you keep the keys for those safes? Or their access code?
Just curious :-)
Re: (Score:2)
In a sufficiently large organization you should have enough areas.
For example, in my organization we keep the combos for other safes in safes with the same sensativity level because, well, drilling is slow and expensive. In different buildings, by preference.
Said combo comes with a list of who's allowed to pick it up, and is in a tamper evident container. Beyond that chain of command comes into play.
Keeping parts of the code in different safes would be even more secure.
My advice in the past (Score:3, Informative)
First, all security experts will tell you that you should not keep copies of that stuff around.
Second, that's not a realistic expectation, stuff happens. The IT guy goes on vacation, has an accident, or dies. (Seen all 3 numerous times.) You fire the Admin for some reason. This building burns down. Etc.
A reasonable thing to do, is keep a password/key log with that critical information that is kept up to date at all times. You have two copies of it. Both are kept secure in good quality safes (not a $200 lockbox).
Both safes are in different physical locations, at least separate buildings, preferably miles apart.
The reason for this is pretty easy. Once again, things happen. I've seen buildings burnt down, flooded, inaccessible due to chemical hazards from a truck wreck, etc. You don't know what will happen, but if you have them stored at separate physical locations, you at least know you will be able to get to one of them if you need to, assuming nobody uses a nuke.
It all falls under that old techie saying, "So, when did your data become important to you? Before or after you lost it...".
Spoonerism (Score:2, Funny)
I literally read that as scoured porn...
Does German work like English? (Score:3, Funny)
Re: (Score:2)
Notice how the Gematik spokesman never actually denies that they didn't want to pay for a backup. A better translation:
"The service provider offered a backup service but we didn't want to pay for it. But they lost the key, so even though they warned us of this and we still said no, it's their responsibility."
Re:An HSM That Requires Continuous Power? (Score:4, Informative)
Don't blame the cards for the stupidity of their administrators.
Re: (Score:2)
For the record, the CIA triad is "Confidentiality, Integrity, Availability", which is actually more applicable in this case. Just sayin'.
Re: (Score:2)