Hacker Increased Chemical Level At Florida City's Water Supply, Police Say (wtsp.com) 117
An anonymous reader quotes a report from WTSP: hacker gained access to Oldsmar's water treatment plant, bumping the sodium hydroxide in the water to a "dangerous" level, according to Pinellas County's sheriff. In a press conference Monday, Sheriff Bob Gualtieri said his deputies, along with the FBI and U.S. Secret Service, are investigating the breach as it is unclear if it came from within the U.S. or from a foreign actor.
The incident first occurred on Feb. 5 at the city's water treatment plant when, around 8 a.m., an operator noticed someone had remotely entered the computer system that he was monitoring. It's a system responsible for controlling the chemicals and other operations of the water treatment plant, Gualtieri said. And this time, Gualtieri says, the hacker did more than just remote in. According to the sheriff, the hacker spent up to five minutes in the system and adjusted the amount of sodium hydroxide in the water from 100 parts per million to 11,100.
"This is obviously a significant and potentially dangerous increase. Sodium hydroxide, also known as lye, is the main ingredient in liquid drain cleaners," Gualtieri added. The operator immediately reduced the levels back to the appropriate amount and "at no time was there a significant adverse effect on the water being treated." Even if the operator did not notice the intrusion, the sheriff, Oldsmar Mayor Eric Seidel and City Manager Al Braithwaite all noted several fail-safes and alarm systems are in place to flag issues of this kind. Gualtieri reinforced that at no time was the public in danger.
The incident first occurred on Feb. 5 at the city's water treatment plant when, around 8 a.m., an operator noticed someone had remotely entered the computer system that he was monitoring. It's a system responsible for controlling the chemicals and other operations of the water treatment plant, Gualtieri said. And this time, Gualtieri says, the hacker did more than just remote in. According to the sheriff, the hacker spent up to five minutes in the system and adjusted the amount of sodium hydroxide in the water from 100 parts per million to 11,100.
"This is obviously a significant and potentially dangerous increase. Sodium hydroxide, also known as lye, is the main ingredient in liquid drain cleaners," Gualtieri added. The operator immediately reduced the levels back to the appropriate amount and "at no time was there a significant adverse effect on the water being treated." Even if the operator did not notice the intrusion, the sheriff, Oldsmar Mayor Eric Seidel and City Manager Al Braithwaite all noted several fail-safes and alarm systems are in place to flag issues of this kind. Gualtieri reinforced that at no time was the public in danger.
The end of the world (Score:4, Funny)
Re:The end of the world (Score:5, Insightful)
Although there was much worry about the world ending due to insufficient quantities of food, the effects of climate change, or even the horror of global nuclear war, humanity was ultimately destroyed by a bored teenager breaking into insecure computers computers that controlled the wheels of society.
Yeah, or we "smart" adults could maybe realize these kinds of systems should not be remotely accessible.
Re: (Score:3, Insightful)
Using good firewall rules, data diodes, and physical separation requires more Ethernet cables and switches! And it requires somebody to figure out how to burn updates to a disc and apply them that way! Do you think the operations team is made of money, huh?
I mean, some systems have the excuse that they operate over lots of different sites, and so they need some kind of quasi-private WAN. This one doesn't seem to have that excuse. Even if the water treatment plant can't afford a good IT security person t
Re: (Score:1)
Someone needs to decide it's a critical resource and protect it accordingly.
Re: (Score:2, Interesting)
Critical according to what definition?
Buses can be used to commit vehicular homicide. Trains can be derailed, potentially spilling toxic materials. Pharmacies and hardware stores store very dangerous chemicals. Are those critical resources that should be protected "accordingly"? Where do you draw that line?
What someone needs to do is a proper risk assessment, and apply security controls based on a rational analysis of the costs and benefits of various measures. Maybe that means that they give up the ab
Re:The end of the world (Score:5, Insightful)
Yes, anything that can kill people should be adequately protected. 2FA is not the answer to something that can fucking kill people. It's don't connect it to the bloody Internet in the first place.
Re: (Score:2, Insightful)
Critical according to what definition?
This is WATER we're talking about here. How about "necessary for life" as a definition?
What someone needs to do is a proper risk assessment, and apply security controls based on a rational analysis of the costs and benefits of various measures. Maybe that means that they give up the ability to let someone remotely diagnose problems or tweak the system, in order to prevent this kind of attack in the future. Maybe it means they use multi-factor authentication on their remote access solution, or that they stop using shared account/password combinations, or something else. Probably it means they make a large set of changes, some large and some small. But slapping a "critical resource" label on it is mostly a way to avoid actually thinking.
"Hurr durr, let's do risk assessment to tell us if water really is that critical!"
Lemme guess, you're a consultant.
Re: (Score:2)
Buses and trains definitely should not be controllable over the internet.
If the pharmacy and hardware store get robots, they shouldn't be internet controllable either. For that matter, neither should grocery store robots unless you want them to lock the doors and mix all the bleach and ammonia.
Re: (Score:2)
> the county or city should be able to afford it and share that resource across departments
You have obviously never worked in government.
Re: (Score:2)
> the county or city should be able to afford it and share that resource across departments
You have obviously never worked in government.
It's called risk mitigation.
Let's review the town of Flint about their hindsight regarding proper water treatment, shall we?
On August 20, 2020, the victims of the water crisis were awarded a combined settlement of $600 million, with 80% going to the families of children affected by the crisis. By November, the settlement grew to $641 million. In January 2021, Snyder and eight other officials were charged with 34 felony counts and 7 misdemeanors—41 counts in all—for their role in the crisis. Two officials were charged with involuntary manslaughter.
Re: (Score:2)
the county or city should be able to afford it and share that resource across departments
This is Flori-DUH, that's not likely to happen. Additionally Oldsmar is a suburb of Tampa/St. Petersburg, where brains go to die.
Re: (Score:3)
> doesn't seem to have that excuse. Even if the water treatment plant can't afford a good IT security person to help full time
They *could* have got me for a day to tell them what's wrong, a day to help with questions about exactly how to solve it, and later a day to sign off on the new systems.
Seek help (Score:3)
Your obsession with me is unhealthly, dude.
Funny thing that surprises people - you actually *can* get in touch with such people. I *have* called in Neil Brown to consult. They are people, not Gods. (You can see the results of that on your own system by running "rpm -q --changelog kernel"). Or you're more the clicky type, pull up the changelog on kernel.org and ctrl-f, then enter my name. Yep, sure enough, Neil Brown was happy to help.
They are just people, and yeah you can email them. Here's me, Dan J
Re: (Score:2)
Ps, if you thought up about four other names to go along with Vint Cerf, you'd probably end up naming someone who just happens to be here on Slashdot, under a nickname.
Slashdot is a place old nerds hang out, so Vint just might be here. And you might have told him that he's an idiot, acting like YOU know everything and everyone else is stupid.
Re: (Score:2)
Seriously get some help dude. Before you're in the news for doing a mass shooting.
Re: (Score:2)
You obsession with random strangers is a problem.
When you're ready, you can call 1-877-726-4727 for help.
Re: (Score:2)
Using good firewall rules, data diodes, and physical separation requires more Ethernet cables and switches! And it requires somebody to figure out how to burn updates to a disc and apply them that way! Do you think the operations team is made of money, huh?
Rather ironic I'm driving many miles today to perform DVD-driven updates on standalone systems. Perhaps they're not "made of money", but should be wise enough to understand risk. Needless to say, a company isn't going to find much sympathy from me when the billion-dollar lawsuits fly after a hacker kills half a damn town poisoning the water supply because you didn't want to spend $20K on one-time infrastructure costs and hire a $50/hour part-time contract worker to do the onsite maintenance.
Hell, this is
Re: (Score:2)
Your reaction seems a little overblown, considering that the fallback security control -- an on-site human operator in the loop -- quickly detected and corrected the attack. Human lives were never at risk.
Maybe a more capable attacker would have been able to compromise that on-site operator's status displays, so that they did not show the elevated levels of dangerous chemicals. Maybe there are other security or safety measures that protect against that kind of thing. They should still remedy the security
Re: (Score:2)
Re: (Score:2)
It isn't hard to understand why it happens, though. The cost savings of having contractors have remote access are a real lure to the people who make the decisions, who are generally money people, not technical people.
That's how Target got hit, after all.
(This was apparently though TeamViewer?)
Re: (Score:3)
IIRC, the Target hack came in through an HVAC vendor. It could have been prevented by isolating the HVAC from the POS (can't imagine why a cash register needs to talk to the thermostat).
It didn't even need to be physical isolation, properly configured VLANs could have prevented the hack.
Re: (Score:2)
You are absolutely correct on this: "Yeah, or we "smart" adults could maybe realize these kinds of systems should not be remotely accessible." I have worked in a water filtration plant (as a chemist, not an operator) and was impressed at the level of monitoring being done by the operator. Many times, automated systems produce massive complacency. In another surprise, what Gualtieri said was correct. An overfeed for that short of a time would be quickly diluted to safe levels.
I
Re: (Score:3)
What reason would this system have to be capable of being accessed by the internet?
Re: (Score:2, Insightful)
TFA says the remote access capability was regularly used so people can diagnose problems remotely. That makes some sense, although one would think that remote users should be restricted to equipment that only has read-only access to the system. But maybe their control systems do not support that kind of separation, so it would require extra equipment to add that security protection.
Re: (Score:1)
When troubleshooting, it's frequently helpful to be able to change things rather than just look at them. You could depend on having an on-site user interacting with the troubleshooter to do the hands-on changes, but that's still cumbersome.
Re: (Score:3)
When troubleshooting, it's frequently helpful to be able to change things rather than just look at them. You could depend on having an on-site user interacting with the troubleshooter to do the hands-on changes, but that's still cumbersome.
How about keeping competent personnel on hand at critical infrastructure sites capable of handling problems on site without exposing the system to hackers?
Re:The end of the world (Score:4, Insightful)
If 100 ppm is normal and 11,100 ppm is toxic, it should not be possible to set the system to 11,100 ppm.
Even if were to turn out necessary to set the lye level up to toxic levels for cleaning or some other relatively rare reason, it would not be necessary to do that remotely.
Re: (Score:2)
Re: (Score:1)
Ah, but what if you could override the system tolerances? Or changing the water flow so that the low levels of additives get added multiple times? Or adjust the inventory records to fool the system into thinking the additives are 0.1% of their actual toxicity?
Even with the "don't let people do that" option you're suggesting, once the system is compromised all bets are off.
I do though accept that a simple unauthorised account access is still restricted to default behaviours for that account, so I would still
Re: (Score:2)
Re: (Score:3)
I heard SolarWinds makes a good system for that.
On second thought, no... disconnect it from the internet permanently. Regardless of Covid, the Black Death, aliens, or anything else. The water supply is one of the few things that needs to be kept secure and running in these events. For perspective, here in TX we have strip clubs that stayed open at full capacity because they claimed to be "restaurants"... I don't imagine Florida did any better.
Re:The end of the world (Score:5, Insightful)
1) People want to work remotely. Not necessarily at home, but in the office and not in the actual water processing plant. Especially if you have 100 controls which are at 100 different sites and would otherwise have to pay someone to drive to them all regularly. Money gets saved. And money is the biggest motivator.
2) Even computer oriented companies have great difficulty hiring exceptional security experts. Why would a water plant, factory, or other industrial site do better at computer security? After all, everyone connected dutifully to the cloud just like Microsoft, Amazon, and Google told them to. What can't be handled in the cloud they will outsource to someone else. Money gets saved!
3) It's like asking why the water company hired Barney Fife as a security guard instead of getting professional war-trained mercenaries to guard things. I mean the citizens would revolt if they found out their fees were being spent on expensive guards, expensive computers, expensive consultants, expensive workers, etc.
Re: The end of the world (Score:2)
2) this is an easy to fix problem: train security engineers
When did training your employees become everyone else's responsibility? There's no labor shortage; there's a competent management shortage.
The end of precedent. (Score:2)
When did training your employees become everyone else's responsibility?
Since some are convinced that one should do things for the love and not the money. Hereafter artists and open-sourcers took this to heart working and learning on their dime, giving their efforts to "everyone else"'s .
Re: (Score:1)
Boss: "We need remote access to that facility for operational and cost control reasons."
Diligent employee: "It's going to be a security risk. Can we train up some security people?"
Boss: "Nah, we're a fucking water company. We know shit about this. Hire in an external to get this set up for us."
The company has thus done absolutely nothing wrong by not training their own security staff.
The fact that the issue was spotted and addressed actually demonstrates that they had very good operational procedures (or an
Re: (Score:2)
0) 24/7 monitoring was required but 24/7 staffing payroll wasn't provided.
Re: (Score:2)
More or less the plot of War Games.
Well, that's rather chilling, isn't it? (Score:2)
Re: (Score:1)
I am reminded of this quote from hackers...
"What are you, stoned or stupid? You don't hack a bank across state lines from your house, you'll get nailed by the FBI. Where are your brains, in your ass? Don't you know anything?"
I have an idea (Score:3, Insightful)
Re: (Score:3)
So the next time that system is out of variance it has no way of throwing an alarm? You might be surprised by how much infrastructure equipment is in isolated buildings in the middle of nowhere that actual humans visit once in a blue moon. There is a control building for the water system in the village where we have our cottage, I've noticed a blackberry vine grown across the doorway for the last year so no one has actually been in the building that long. I've worked at an electrical utility where they m
Re: (Score:2)
If all you need is monitoring then point a webcam at it. Seen that done before.
Re: (Score:2)
Botanical security (Score:2)
Blackberries grow on a cane, which has wicked thorns that can rip your skin to shreds.
That is if what you see is indeed a blackberry plant and not the vine of something else. In that case, the building is much less secure.
Re: (Score:2)
We live in the Pacific Northwest, we're much better acquainted with blackberries than anyone really wants to be. I paid my rent one month hacking the blackberry canes that had taken over the landlord's property (got really, really good with a machete).
Re: (Score:2)
Blackberries grow on a cane, which has wicked thorns that can rip your skin to shreds.
When being pedantic about terminology it would seem sensible to use the correct terminology throughout your statement:
"First- and second-year shoots usually have numerous short-curved, very sharp prickles that are often erroneously called thorns."
The real name for those sharp things (Score:2)
There is a name for those sharp things on the blackberry plant that I call them by when I go picking, but in deference to our younger members, I won't use that term here.
Re: (Score:2)
Yes, it's an "aggregate drupe". Tomatoes on the other hand are berries, apparently.
Re: (Score:2)
Nice in theory, not so much in the real world. I've worked in utilities, they don't make communications and monitoring decisions lightly. Even in Flori-DUH.
Re: (Score:2)
Well, I take back my previous statement, should never have included FloriDUH in that statement. Turns out they were using Team Viewer on the machine with a shared easily-guessed password.
Nice (Score:2)
Aggressive water in a region where they are still having lead pipes could be very dangerous.
But luckily this is Florida, where the water is disgusting anyway and people don't even use it to make ice, much less drink it.
It's that free water on the table that every customer was entitled to, dear kids, in the olden times, when we still met in places called 'Restaurants', I remember it fondly.
Re: (Score:2)
This utility is connected (Score:4, Insightful)
Re: (Score:1)
IOCT: Internet of Compromised Things. My kids will face a strange strange future (besides their father).
Re: (Score:1)
A strange future like it was 20 years ago, when everything wasn't "connected" and things went along fine.
Re: (Score:2)
I’m sure they will fire the guy who sweeps the floor.
Re: (Score:2)
Would being disconnected be any better?
They aren't going to pay someone to monitor it 24/7 so disconnected just means more vulnerable to undetected faults and physical tampering.
It might be perfectly secure, the hack could be an inside job or someone in their main network.
Re: (Score:1)
Re: (Score:1)
Re: (Score:3)
Would being disconnected be any better?
Yes
They aren't going to pay someone to monitor it 24/7
It's the city water grid so why not? The building is going to be occupied and running 24/7 so should the operator's seat.
so disconnected just means more vulnerable to undetected faults and physical tampering.
It might be perfectly secure, the hack could be an inside job or someone in their main network.
The controls shouldn't be accessible from their main network either, ideally the whole control system is air-gapped.
It doesn't mean 100% secure, you can still have disgruntled employees or some hacker team from a Hollywood movie showing up, but it makes attacks much harder. [mcafee.com]
But if you're online you're basically exposed to the entire Internet. Even to malicious hackers who are smart enou
Re: (Score:2)
The whole thing is silly anyway. Anything "important" like this should have multiple levels of redundant checks from different, disconnected systems. You raise level to 11,100? Oops, rejected - value seems weird. If you don't like it, go to this other system completely firewalled and change the rules. OK, you did that? Great. Uh oh, the downstream check failed - value of of bounds error. Better send someone down to the basement to force and allow a manual override.
Whew, OK I got my 11,100 level in. Oh. Shit
Re: (Score:2)
The whole thing is silly anyway. Anything "important" like this should have multiple levels of redundant checks from different, disconnected systems. You raise level to 11,100? Oops, rejected - value seems weird. If you don't like it, go to this other system completely firewalled and change the rules. OK, you did that? Great. Uh oh, the downstream check failed - value of of bounds error. Better send someone down to the basement to force and allow a manual override.
Whew, OK I got my 11,100 level in. Oh. Shit. Now the pre-dispensing system is complaining. Someone will have to go and push some buttons to allow this unusual condition. Send John. OK, done! Well, except now the dispensing system is hard-limited to 300. Shit, now we have to go in with a fucking special screwdriver after we've unlocked the chassis and make 10 adjustments.
My guess is that while it's probably not this hard (it should be), in reality that hacker didn't get as close as he thought. And now he's going to reap the whirlwind, they won't fuck around in trying to find them.
The article isn't clear about how the "11000" was set. Was it a target value written directly to a PLC that would start injecting chemical until it hit the target value or was it a value the SCADA system itself used to manage the injection? (no reason to assume the device that adds the chemical is also the device that can measure the levels).
The problem with all those checks you propose is that these systems are both heavily customized yet still need to be ridiculously reliable (don't want a bug writing 111
Re: (Score:2)
They won't pay someone to monitor it because you won't stand for that added cost on your bill/taxes.
Air gaps won't stop someone controlling the system. If you put a minimum wage operator there they will do what they are told via spoof email or proof phone call. It's probably even more vulnerable, now they don't even need to hack your network, just get an email address or phone number.
Re: (Score:2)
Dictators at work? (Score:1)
If this is a foreign gov't in action, it can be considered an act of war, and a war crime.
Re: (Score:3)
Re: (Score:1)
No, I mean an enemy nation hacking into US infrastructure and screwing with it. Yes, the builders connecting it to the public internet is dumb, but two wrongs don't make a right.
Forgetting to lock your front door is not a sufficient reason to dismiss your murder by a robber.
Re: (Score:2)
Does no one know what "remote in" means any more? Holy carp. This used to be a techie site, what the frack happened?
They didn't connect the water plant directly to the Internet, some things are too stupid even for Flori-DUH. Someone connected to their network, almost certainly over a VPN connection. Once they had done that they have access to everything on that LAN. I'd be very surprised if this doesn't turn out to be someone's kid who got playing around when their parent forgot to log out, saw an inte
Re: (Score:1)
Remoting in over the internet is still "connecting to the internet". Since land lines are dying, almost any remote service of any kind pretty much is forced to use the internet. (And land lines were hackable also, ask Woz.) Maybe there are ways to completely avoid anything touching the internet, but it would likely be expensive or inconvenient.
Re: (Score:3)
Power system SCADA networks are supposed to be air gapped from the rest of the great wide world, including corporate LANs, and almost all of them are. And you're right, it's expensive and inconvenient but the risk coming from NOT doing that is to large.
At least ... (Score:5, Funny)
At least the drains would have been clean and free of obstructions.
'"Sodium hydroxide, also known as lye, is the main ingredient in liquid drain cleaners," Gualtieri added.'
Re: (Score:2)
You drink it first.
My guess is? (Score:2, Insightful)
I bet the system that was breached was just a Windows PC running remote control software like TeamViewer.
The way packages like that work, there's usually at least one paid account/login that's owned by whoever in I.T. does the remote support, and once you sign into it - the app gives you a list of all of the workstations that are configured for one-click remote unattended access.
Obviously, it CAN be configured in much more secure ways. But this is yet another situation where ease of use/administration plays
"Hacker"? (Score:2, Interesting)
Re: (Score:2)
yeah right (Score:3)
why should i believe the water treatment plant? these guys literally lye for a living.
Re: (Score:2)
why should i believe the water treatment plant? these guys literally lye for a living.
Water you talking about?
Re: (Score:3)
Such basic humor.
Not a hack (Score:5, Interesting)
I'm betting there is no hack. Just a fat-fingered operator. You don't change 100 to 11,100 as a hack. That looks much more likely to be a key repeat issue.
Re: (Score:2)
Fat-fingered hacker?
But your explanation is much more plausible and accounts for a common human behavior - lie about the lye. Your interpretation will be the correct one.
Re: Not a hack (Score:1)
It's intereting how people like you are willingly black-eyed and always choose the most comforting and harmless possibility, regardless of having amy knowledga making any of the choices more likely.
Like conspiracy theorists always choose the most out there one.
You both just pick what reaffirms what you want to be real. It's just that they need the crazy fantasy to explain that they can't handle the world, and you need the comforting fantasy because you can't handle the crazy of the real world.
I'm sorry, Occ
Re: (Score:3)
It's intereting how people like you are willingly black-eyed and always choose the most comforting and harmless possibility, regardless of having amy knowledga making any of the choices more likely.
Like conspiracy theorists always choose the most out there one.
You both just pick what reaffirms what you want to be real. It's just that they need the crazy fantasy to explain that they can't handle the world, and you need the comforting fantasy because you can't handle the crazy of the real world.
People are not gods. Everyone as a matter of necessity has to make decisions based upon incomplete information. There is nothing wrong with having an opinion based on little to no evidence. It is when you take that extra step of disregarding the fact the opinion you formed has a tenuous evidentiary basis that you get into trouble.
Lets consider your own words "people like you are willingly black-eyed and always choose the most comforting and harmless possibility" followed by derisive commentary. Here you
Re: (Score:2)
While there is some truth to what you say, it's a non sequitur. No one invoked Occam's razor. They just used subjective probability and picked what they think is the most likely/plausible cause. You could argue against their assessment, but I'm cynical enough to think there is a decent chance they are right.
Re:Not a hack - a stuck key (Score:2)
Stuck key more likely. Ever seen computer "type by itself" when a key gets stuck? Spooookey...
Re: (Score:2)
Re: (Score:2)
Why aren't they using Dominion Voting's Security? (Score:1, Offtopic)
We've been told it's unhackable [youtube.com]
.
The billion-dollar lawsuit (Score:2)
for thinking that thought is headed your way.
Re: (Score:2)
sanity checks (Score:1)
Re: (Score:2)
Re: sanity checks (Score:2)
Maybe the sanity override was available remotely too?
Or, more likely, the interface did not allow such values, but tje HTTP POST interface (or equivalent), *did*. :)
Re: (Score:2)
Re: (Score:2)
Our sodium hydroxide goes to eleven.
pH (Score:2)
Please let it be fluoride, please let it be fluori (Score:2)
*crosses fingers*
*drinks pire medicinal alcohol and rain water*
*starts reading TFS*
Florida Man (Score:1)
Trusting compromised systems (Score:2)
We changed the setting back and kicked the intruder... mission accomplished...
11,100 (Score:2)
"adjusted the amount of sodium hydroxide in the water from 100 parts per million to 11,100": afaik, he did NOT. He apparently changed the set point to that, but it seems improbable that the system could have physically achieved that set point, and probably couldn't even come close. It would have taken extreme over-engineering of valves, pumps etc. to be able to put that much NaOH into a system when it was supposed to be two orders of magnitude lower.
Re: FLINT MICHIGAN (Score:1)
I thought they were talking about a place in the Flintstones!