Hackers Are Holding Baltimore's Government Computers Hostage (gizmodo.com) 172
On May 7, hackers infected about 10,000 of Baltimore city government's computers with an aggressive form of ransomware called RobbinHood, and insisted the city pay 13 bitcoin (then $76,280, today $102,310) to cut the computers loose. The hackers claimed the price would go up every day after four days, and after the tenth day, the affected files would be lost forever. From a report: "We won't talk more, all we know is MONEY!" the ransom note read. "Hurry up! Tik Tak, Tik Tak, Tik Tak!" But the city has not paid. In the two weeks since, Baltimore citizens have not had access to many city services. The city payment services and email systems are still offline. A May 7 Baltimore Sun report stated the Robbinhood ransomware used in this attack encrypts files with a "file-locking" virus so the hackers can hold the files hostage. Among the departments that have had issues with their email and phone systems are the Department of Public Works, the Department of Transportation, and the Baltimore Police Department.
According to the Wall Street Journal, Baltimore Health Department's epidemiologists aren't able to use the network that allows them to alert citizens of certain which types of drugs are causing recent overdoses. Many services have resumed through phone, and vital emergency systems like 911 and 311 reportedly continued to function. The ransomware froze the system the city uses for executing home sales, which reportedly hurt the local market, but the city began implementing a manual workaround earlier this week.
According to the Wall Street Journal, Baltimore Health Department's epidemiologists aren't able to use the network that allows them to alert citizens of certain which types of drugs are causing recent overdoses. Many services have resumed through phone, and vital emergency systems like 911 and 311 reportedly continued to function. The ransomware froze the system the city uses for executing home sales, which reportedly hurt the local market, but the city began implementing a manual workaround earlier this week.
Re: (Score:3, Insightful)
I don't understand what the problem is. Why doesn't the city of Baltimore lock down the security hole and restore from backups?
They DO have backups, right?
Re: (Score:1)
On the enterprise, it isn't difficult, but takes a little bit of budget:
1: Take admin access away from all user accounts. If a user needs admin rights, give him a second user with those privs, only used for those functions. This applies from the top on down.
2: All domain admins get a PAW/SAW. Domain accounts only get logged on there, or a DC. If action needs taken on a box as admin, they can use a separate account with admin rights, ensuring that the domain is protected.
3: AppLocker and FSRM are used
Hackers like this (Score:5, Insightful)
need to be found and killed.
Re:Hackers like this (Score:4, Insightful)
Governments who don't make backups, and therefore enable hackers like this, need to be found and killed.
Re: (Score:3)
Well, considering this is in Baltimore City -- the odds are pretty good for that to come to pass.
Re: (Score:1)
Roddy McDowall or Charlton Heston?
Damn dirty hackers!
Re: (Score:3, Insightful)
Unless you have a cold backup of every single system on every single day (which would be expensive as hell), if your backup is infected restoring it won't help you.
You would literally need discrete backups for each day, and the ability to know exactly when you got infected ... and then when you restored you still have the same vulnerabilities in place as before, and will get hacked almost immediately since you have the exact same weaknesses. For all you know, you were hack
Re:Hackers like this (Score:5, Insightful)
You would literally need discrete backups for each day, and the ability to know exactly when you got infected ...
A decent rsnapshot backup scheme takes care of this without requiring huge amounts of disk space for each day - and the fact that a particular day's backup jumped WAY up in size (since rsnapshot backups usually are saving just the changed files) should alert a decent IT group to the possibility of a problem - like the day that the ransomware hit.
You might end up being out some recently changed files, but it's likely the vast majority of files will not change over the course of months or even years. Unless the malware has been on the system and dormant for many years, you should be able to mitigate this.
Re: Hackers like this (Score:1)
Exactly. Ransomware encrypting data will lead to a massive delta change in the next backup job.
Re: (Score:1)
Which tells you when the payload activated, not when the original infection happened, which if it cannot be determined. And if you can't locate and neutralize that you will run the risk of a repeat occurrence.
Re: (Score:1)
Daily backups of every service, sure. Not every pc - a user's pc can be reformatted. (Lost anything? Should have stored on the server, not the local disk, you moron.)
Daily backups is not that expensive, of course you use an incremental system so the stuff that doesn't change don't need space. Which also tells you exactly when the encryption started, because that is when the incremental backups suddenly got much bigger. But you already know exactly when encryption started - because services crashed then. Dat
Re: (Score:3)
Don't forget, the governments get heavily criticized if they spend money on computer security. They get heavily criticized if they spend money on anything. If corporations get these kind of hacks when they make a profit, then you gotta expect that a municipality running on a shoestring budget and a mantra to do more with less will hit security issues as well.
Re: (Score:2)
Re: (Score:2)
But that's easy. A relatively modern storage array can take snapshots. Just have a sane retention period on that to start with and yes; you can recover. And honestly it's not that hard to tell when you're infected because these sorts of ransomware don't really wait around; their payloads activate pretty swiftly after an infection because they need to get their profit quickly. If they don't they run the risk that their virus will be found or their C&C network will be shut down (if there is one of course)
Re: (Score:1)
"Ransomware is only able to exist because of blatant incompetence."
FYI, Baltimore City government is almost 100% run by Blacks. What a surprise, ehh?
Moderated -1: Uncomfortable truth
Re: (Score:2)
Mod parent up.
They need to be hunted down and killed, and the action publicized to warn anyone else thinking of doing this.
I'd contribute to a Kickstarter for this, if there was one.
A test of NSA capability (Score:5, Interesting)
This is an opportunity to see if that Utah data collection center (https://en.wikipedia.org/wiki/Utah_Data_Center) is usable for an actual national security task. Have it find and expose these crooks, so we can have local operatives terminate them in some specular and messy enough way that ransomware criminals are permanently dissuaded from pulling this kind of attack again.
Re: (Score:2)
terminate them in some specular and messy enough way
You mean like Domestic drone strikes and black bagging to Gotmo?
Re: (Score:3)
Public drawing and quartering would suffice.
Re: (Score:2)
This is an opportunity to see if that Utah data collection center (https://en.wikipedia.org/wiki/Utah_Data_Center) is usable for an actual national security task.
They've been too busy collecting people's phone records and Facebook timelines to get involved with helping local governments out of a jam.
Re: (Score:2)
Re: (Score:3)
Considered in isolation, an attack on one city is not national security, but the threat of ransomware considered as a whole certainly is. Entire national health systems, like the British NHS, have been brought down by it. It's cyber-warfare by a substate actor, and needs to be responded to as such. If that means strangling Russian gangsters with their own intestines and melting down the Bitcoin market, so be it.
It's all very well to flame Baltimore for not adequately backing up its systems, but it's like bl
Re: (Score:3, Informative)
Well, the main contributing factor was probably:
a) They're Baltimore
Re: Who did they hire? (Score:2)
Backups?... (Score:5, Insightful)
How the heck can a city as big and as well run as Baltimore not have backups? This is not rocket science here. It is damn trivial to have ransomware resistant backups. $6/PC/month backs up home machines. Veeam is dirt cheap and is great for the enterprise, with ransomware resistance if tapes or cloud storage used. In fact, this is IT 101.
The city needs to follow their DR plan (which is assumed they have), clean up their mess, restore, and get back to what they are doing. If they don't have backups, someone needs to be fired. In any IT shop in the world, not being able to restore == out the door in seconds flat.
Re:Backups?... (Score:4, Insightful)
You beat me to it, that was exactly my first thought, why not restore the backups and be done with it? Yes, it will cost money to restore (IT time mostly to restore the entire network), but it would solve the problem. Of course if they have no functional backup solution, they are SOL now, since it's been more than 4 days and if they hackers followed up on their promises with erasing the keys, even paying the ransom is no longer an option.
Re:Backups?... (Score:4, Insightful)
There's a lot more work than just restoring backups...
If you roll back to a state from before you got infected, then you're vulnerable to however you got infected.
There's also no telling how long the attackers were on the network before they triggered the encryption process, you could end up restoring backups which are already infected.
Once someone has got into a network they can also acquire huge amounts of information from it, for instance passwords for every user and every system - if you don't completely rebuild, there will usually be information which allows continued access.
Re:Backups?... (Score:4, Insightful)
Re:Backups?... (Score:5, Insightful)
Veeam is great, but no where near dirt cheap. I would say hella expensive actually.
For instance, for 70 VMs and 3 servers, just veeam backup is 12k per year USD. I am not sure if a city government is more or less resources that need backing up than that.
https://www.veeam.com/pricing-... [veeam.com]
and that does not include the million other little licenses that one needs. Many things are sold per cpu socket on the cluster as well. And then you need a storage array that can retain for at least a few weeks of backups. And a DR site to mirror everything to, plus bandwidth, network connections, co-location fees.... Now it starts to look like a real IT capital project and less like 'dirt cheap' as you make it seem.
Is it 100% necessary and a justifiable expense? you bet! but don't pretend it does not cost money to do things properly.
Re:Backups?... (Score:4, Insightful)
So the city goes and proposes to hire a new IT security expert. They get voted down by the board because it costs too much money that they don't have. They suggest raising taxes and then that's voted down because it might hurt their election chances. So they stick with what they have. And that's how you get a dysfunctional IT team in a municipal government.
And from what I can tell from the story, they are bringing stuff back up. It just takes time.
Re: (Score:1)
Re: (Score:2)
So the city goes and proposes to hire a new IT security expert.
That may not solve the problem anyways. While there are numerous "Security Experts" who are incompetent, the most likely source of the problem is the management not making the changes needed to be actually be more secure.
Servers with unpatched vulnerabilities from over a year ago? Hey, that can't be solved.
Problems with distributing third party patches? That can't be solved.
Municipal governments are all about setting things in up in such a way as nobody can be held accountable. It is so political in nature,
Re: (Score:2)
For instance, for 70 VMs and 3 servers, just veeam backup is 12k per year USD. I am not sure if a city government is more or less resources that need backing up than that. https://www.veeam.com/pricing-... [veeam.com]
That's retail. State and Local governments usually pay 1/3 to 1/2 of that amount. And what you're missing is that the VMWare licensing on those three hosts would run at least 100k retail, plus (assuming you're running Windows) another about 60k for some Windows Datacenter licenses. By comparison, the 12k for a backup solution is a rounding error.
Re: (Score:2)
How the heck can a city as big and as well run as Baltimore not have backups?
Apparently they're not well-run.
Re:Backups?... (Score:5, Informative)
Because it's Baltimore. The city that has stacked decade upon decade of failure and systematic destruction of its tax base, corruption, and sheer incompetence. You know, the most dangerous city in the USA [amren.com] with the highest per capita murder rate in the country. The city where two of the last three mayors have resigned [nytimes.com] in disgrace [baltimoresun.com] (keeping their pensions, of course), while the third declined to run for re-election after telling the police to "stand down" during the 2015 riots [washingtonpost.com], while giving rioters "room to destroy." [cbslocal.com] The city so broke and mismanaged that 90% of its annual operating revenue comes from the Federal gov't or elsewhere in the State of Maryland.
Why is anyone even remotely surprised that this happened in the shithole known as Baltimore?
Re: (Score:2)
Re: Backups?... (Score:1)
You so angry. Here, have a lollipop.
Oh shit (Score:5, Funny)
Without those computers, Baltimore might turn into a lawless shithole.
Re: (Score:1)
Hmmm a link to Fake news from the fake news king
Re: (Score:1)
You have Trump on the brain. Enough already.
Disposable infrastructure (Score:5, Insightful)
Let's all learn the right lesson here, folks...
If you have important infrastructure, make sure that any part (or all) of it can be rebuilt from scratch, using offline resources. Have fresh backups of your data, but your infrastructure should be deployable from cold media, and in that deployment process, it should automatically download & install patches, so you don't just get reinfected as soon as your new infrastructure comes online.
No, your processes are not special enough to justify keeping that old Java 1.4.2 web application around, and no, you can't keep running Windows XP just because you "know it is compatible with the software". If your disaster recovery plan doesn't have a procedure to rebuild everything to a clean & secure state, you're neglecting a significant risk.
Re: (Score:2)
No, your processes are not special enough to justify keeping that old Java 1.4.2 web application around
Unless of course, that application is your main revenue generator.
Re: (Score:3)
So of course you're doing active development on your main revenue generator, and you're routinely upgrading it to run on a supported Java version, right?
No? Well, that's where that "neglect" word comes in... Whoever made the decision that saving money on development was more important than replacing a rusted-out piece of critical production equipment has made a decision on risk, and should be held responsible for it.
Not possible with integrated Windows Update etc. (Score:1)
Systems need to be constantly updated in order to be secure. They can never stand still.
So that backup you made is quickly out of date. It cannot be made live until all the updates and patches have been applied.
More importantly, the whole thing is too complex to restore in any case. The idea that you can just load a tape onto a disk drive is so 20th century. You now need to run complicated installation procedures that reestablish protocols between many tightly integrated but distinct systems, some of whi
Re: (Score:1)
So just pay the ransom. It is much cheaper in the end.
Dear god, no. What happens when the building catches fire and burns to the ground? Who do you pay the ransom to in that case?
Disaster recovery is all about recovering from ANY (conceivable) disaster. This means write-only, offsite backups and a regularly-tested process for restoring from backups. Also nice would be a contingency plan to run in a degraded state during the recovery (paper forms, telephone banks, etc.). Any data infected by ransomware should be treated as permanently lost (accidentally de
Re: (Score:2)
So of course you're doing active development on your main revenue generator, and you're routinely upgrading it to run on a supported Java version, right?
No? Well, that's where that "neglect" word comes in... Whoever made the decision that saving money on development was more important than replacing a rusted-out piece of critical production equipment has made a decision on risk, and should be held responsible for it.
Oh yes, active development is taking place - on an in-house replacement system that will more truly reflect our business processes. As proposed and pushed by management.
But you see, management wasn't keen on funding additional FTEs for such work, and they also argued among themselves on how the development should proceed, all while the development was taking place by those overworked and underappreciated employees who truly understood the gravity of the situation.
Re: (Score:2)
your processes are not special enough to justify keeping that old Java 1.4.2 web application around
Unless of course, that application is your main revenue generator.
If you got yourself into that situation then you deserve the Darwin treatment. Behold, the universe unfolding as it should.
Replace Windoze with GNU/Linux (Score:1)
The best solution would be to ditch all non-free software, especially from M$, in favor of free software such as GNU/Linux. Free software is far more secure from the ground up. M$ and non free software vendors just care about money so security matters little to them whereas free software is from ordinary people that care about the software they code. Let this be an expensive lesson for Baltimore to aqvoid non-free software. IF they continue with M$ they will suffer the same fate.
--
Friends don't help frie
Smoking is equivalent to using M$ junk (Score:1)
I am glad we agree on something, non-free software is non-secure. Using M$ junk is the equivalent of smoking, both allow for virii to attack systems whether technologicval or biological. All you did was reiiterate what I posted.
--
Friends don't help friends install M$ junk
Re: (Score:2)
I have plenty of software that will NOT work on anything except 2000/XP because
Have you tried? With all the compatibility shims and virtualization options available today? Those take care of about half the compatibility issues I've come across.
the company is no longer in existence and there is literally no other alternative to the software which I use for the hardware that I have
...so if it stops working tomorrow, your company is screwed, and you've openly admitted that you don't have a disaster recovery plan.
and I'm not dropping another $1,000,000
You can either spend money now or later. For about a quarter of that cost, you can hire a developer for a few years to write your own software that would be compatible with modern operating systems, removing the ri
Re: (Score:3)
Re: (Score:2)
If you read the story, which is light on details, it seems they are restoring and rebuilding. How long would it take your company to restore everything from backup?
Re: (Score:3)
Re: (Score:2)
The largest I've rebuilt personally was about 200 systems, with about 10% being unique servers. It took all night, followed by a day of functional testing.
Now, the caveat is that we did use test data rather than the actual backups of live data, so that came online faster. We were also staged in a test environment, so network equipment was tested separately.
Backup is not trivial, and I'm not suggesting that it's trivial. Backup is vital, and disaster recovery tests should be considered a routine part of runn
Re: (Score:2)
Re: (Score:2)
No, your processes are not special enough to justify keeping that old Java 1.4.2 web application
The security guy can point that out, but the security guy doesn't get to make a decision on it. A lot of times, those kinds of decisions are made by the City Council and they will get roasted alive if they make decisions that make things more expensive or more difficult for the citizens.
Rationally, we all already know what the problems are. Those problems are not easily solvable in a municipal environment.
Re: (Score:2)
It's not just the security guy that needs to learn this lesson, though. The councilman making the decision needs to understand he's not just disappointing a security guy, but he's opening his city to risk. The citizens need to understand that when they fight for tax breaks, they're taking money away from their future safety to put in their pockets now.
Ultimately, that's another important lesson: Security isn't just a bolt-on attachment to existing processes. It is a comprehensive attitude impacted by all as
Re: 2019: the year of Windows on the Desktop? (Score:1)
Call Grandma (Score:2)
She was the treasurer in the 80s, and she knows how to make it all go with paper. Uncle Billy was the city council chair in the 70s, he knows how to make the council go with just paper.
Sorry Mr. City-hall clerk, your computer down right now, and the hardcopies are in the basement. Mr. Mayor needs them 10 minutes ago. You know what to do.
Where are the backups? (Score:5, Interesting)
Turns out the email was fake - generated by a ransomware virus, and the file contained the virus payload. The moment she realized what was happening, she yanked the network cable out of the wall (physically broke the cable) to prevent it from spreading to other computers on the company's network, and pulled the computer's power plug to to stop it from encrypting more files. (Yeah, yeah, turning it off would've been enough, but she was in a panic.) She then called the IT department and told them what happened.
IT popped her computer's hard drive into another computer, and confirmed that it had indeed been encrypted. They pulled a spare computer off the shelf, and restored the latest backup of her system from the previous night. They wiped her computer's drive, and it became the new spare computer. She took the rest of the day off while they did this, and was back at work the next day as if nothing had happened. The impact on her company was the same as if she had called in sick one day, and IT had had to re-image a system which had suffered file corruption.
You're playing with fire if you don't have backups. It doesn't even have to be ransomware - mistakenly formatting a data drive instead of that USB flash drive, a program error, a hard drive failure, or a flood or fire will all cripple you the same as ransomware if you don't have backups.
Re: (Score:2)
Opening an Office document isn't going to automatically start encrypting stuff unless you have macros default enabled. They are not enabled by default; even Microsoft learned that lesson, finally.
What she probably got was a document that displays nothing but a "You must click "enable content" to read this document because reasons." image. "Enable Content" is Microsoft's euphemism for "Please bend over and let me ream you with all my nasty malware macros."
Of course, if she had macros enabled by default, sh
Re: (Score:2)
Why do you think they are not restoring from backups?
"She took the rest of the day off while they did this, and was back at work the next day as if nothing had happened."
That was one fungible machine that could be disconnected from the network while being re-imaged and replaced with a spare on hand to receive the restore.
Now do that for every machine and still keep the organization running without reinfecting the new machines and without having a 100% spare capacity.
Re: (Score:2)
None of this technology is foolproof, if you come to rely on it too much you will eventually get burned. Nothing is a "non issue" and there are plenty of ways around it...
A lot of legitimate companies send out emails that look exactly like phishing, a good phishing campaign will replicate the appearance of these emails. Users who have dealt with such companies may even be expecting similar mails and be more likely to fall for them. Most such mails won't get filtered as the companies sending them would start
Proving old Ben right again. (Score:2)
"If you fail to plan, you are planning to fail!" - Benjamin Franklin
No plans seem to exist for data loss.
How can they have no backups? (Score:2)
Re: (Score:2)
Technical incompetence can also be a result of managerial incompetence too, like hiring the cheapest available staff who will not do a good job, or by not only giving the technical staff a limited budget but also placing constraints on how they can spend it - so they waste most of the budget on management's preferred vendors instead of acquiring better value.
It's possible to build highly secure and resilient systems on a shoestring budget, if you have the knowledge and freedom to do so.
MICROS~1 strikes again .. (Score:3)
Re: (Score:2)
Yeah because a program that has admin rights granted by the user can't do anything to stop a Linux service. derp derp.
Somebody needs to be arrested (Score:2)
Somebody needs to be arrested. You know who? The idiot that decided to run the people's business on a Windows computer.
Cheaper then possible IT Security (Score:2)
Maybe stop screwing up? Maybe do not let IT security be handled by dumb, fat, lazy and incompetent internal employees and maybe ask somebody that actually has a clue how to do things? Yes, that costs money. And yes, that will result in things that need to be fixed, which also costs money.
We will see this more and more. Large organizations are defenseless and helpless and it is all their own fault.
Re: (Score:2)
If they had used unix or novell they would have had to hire expensive highly trained IT staff, they used windows because it's "easy to use" and allows them to hire "dumb, fat, lazy and incompetent internal employees" who are cheap.
This is how microsoft were always marketing, and this story highlights the end result.
Re: (Score:2)
Indeed.
What's New in Baltimore? (Score:3)
The city that took Edgar Allan Poe and gave us Frank Zappa and John Waters?
What's New in Baltimore? - Frank Zappa [youtube.com]
Also, Spiro Agnew, John Astin (Gomez Addams), David Byrne, Cab Calloway, Ben Carson, W. E. B. Du Bois, Mama Cass, Alger Hiss, Billie Holiday, Johns Hopkins, John Kassir (the Crypt Keeper), Eli Lilly, Thurgood Marshall, H.L. Mencken, Ogden Nash, Randy Pausch, Nancy Pelosi, Emily Post, Tupac Shakur and Upton Sinclair.
I'm sure there's plenty for everyone to love and/or hate in that bunch. Quite the cast of characters.
Re: (Score:2)
What nationality uses "tik tak" to represent the sound of a ticking clock? If someone want to play hunt-the-scumbag, knowing that might narrow down the hunting ground.
That sounds good. But people hunting the scumbag have to remember that he might use the idioms of another country, to make it seem like he's in that country.