Follow Slashdot stories on Twitter

 



Forgot your password?
typodupeerror
×
Security

Hacker Increased Chemical Level At Florida City's Water Supply, Police Say (wtsp.com) 117

An anonymous reader quotes a report from WTSP: hacker gained access to Oldsmar's water treatment plant, bumping the sodium hydroxide in the water to a "dangerous" level, according to Pinellas County's sheriff. In a press conference Monday, Sheriff Bob Gualtieri said his deputies, along with the FBI and U.S. Secret Service, are investigating the breach as it is unclear if it came from within the U.S. or from a foreign actor.

The incident first occurred on Feb. 5 at the city's water treatment plant when, around 8 a.m., an operator noticed someone had remotely entered the computer system that he was monitoring. It's a system responsible for controlling the chemicals and other operations of the water treatment plant, Gualtieri said. And this time, Gualtieri says, the hacker did more than just remote in. According to the sheriff, the hacker spent up to five minutes in the system and adjusted the amount of sodium hydroxide in the water from 100 parts per million to 11,100.

"This is obviously a significant and potentially dangerous increase. Sodium hydroxide, also known as lye, is the main ingredient in liquid drain cleaners," Gualtieri added. The operator immediately reduced the levels back to the appropriate amount and "at no time was there a significant adverse effect on the water being treated." Even if the operator did not notice the intrusion, the sheriff, Oldsmar Mayor Eric Seidel and City Manager Al Braithwaite all noted several fail-safes and alarm systems are in place to flag issues of this kind. Gualtieri reinforced that at no time was the public in danger.

This discussion has been archived. No new comments can be posted.

Hacker Increased Chemical Level At Florida City's Water Supply, Police Say

Comments Filter:
  • by alvinrod ( 889928 ) on Monday February 08, 2021 @05:17PM (#61041724)
    Although there was much worry about the world ending due to insufficient quantities of food, the effects of climate change, or even the horror of global nuclear war, humanity was ultimately destroyed by a bored teenager breaking into insecure computers computers that controlled the wheels of society.
    • by geekmux ( 1040042 ) on Monday February 08, 2021 @05:22PM (#61041740)

      Although there was much worry about the world ending due to insufficient quantities of food, the effects of climate change, or even the horror of global nuclear war, humanity was ultimately destroyed by a bored teenager breaking into insecure computers computers that controlled the wheels of society.

      Yeah, or we "smart" adults could maybe realize these kinds of systems should not be remotely accessible.

      • Re: (Score:3, Insightful)

        by Entrope ( 68843 )

        Using good firewall rules, data diodes, and physical separation requires more Ethernet cables and switches! And it requires somebody to figure out how to burn updates to a disc and apply them that way! Do you think the operations team is made of money, huh?

        I mean, some systems have the excuse that they operate over lots of different sites, and so they need some kind of quasi-private WAN. This one doesn't seem to have that excuse. Even if the water treatment plant can't afford a good IT security person t

        • by dagarath ( 33684 )

          Someone needs to decide it's a critical resource and protect it accordingly.

          • Re: (Score:2, Interesting)

            by Entrope ( 68843 )

            Critical according to what definition?

            Buses can be used to commit vehicular homicide. Trains can be derailed, potentially spilling toxic materials. Pharmacies and hardware stores store very dangerous chemicals. Are those critical resources that should be protected "accordingly"? Where do you draw that line?

            What someone needs to do is a proper risk assessment, and apply security controls based on a rational analysis of the costs and benefits of various measures. Maybe that means that they give up the ab

            • by ahodgson ( 74077 ) on Monday February 08, 2021 @06:26PM (#61041978)

              Yes, anything that can kill people should be adequately protected. 2FA is not the answer to something that can fucking kill people. It's don't connect it to the bloody Internet in the first place.

            • Re: (Score:2, Insightful)

              by Anonymous Coward

              Critical according to what definition?

              This is WATER we're talking about here. How about "necessary for life" as a definition?

              What someone needs to do is a proper risk assessment, and apply security controls based on a rational analysis of the costs and benefits of various measures. Maybe that means that they give up the ability to let someone remotely diagnose problems or tweak the system, in order to prevent this kind of attack in the future. Maybe it means they use multi-factor authentication on their remote access solution, or that they stop using shared account/password combinations, or something else. Probably it means they make a large set of changes, some large and some small. But slapping a "critical resource" label on it is mostly a way to avoid actually thinking.

              "Hurr durr, let's do risk assessment to tell us if water really is that critical!"

              Lemme guess, you're a consultant.

            • by sjames ( 1099 )

              Buses and trains definitely should not be controllable over the internet.

              If the pharmacy and hardware store get robots, they shouldn't be internet controllable either. For that matter, neither should grocery store robots unless you want them to lock the doors and mix all the bleach and ammonia.

        • > the county or city should be able to afford it and share that resource across departments

          You have obviously never worked in government.

          • > the county or city should be able to afford it and share that resource across departments

            You have obviously never worked in government.

            It's called risk mitigation.

            Let's review the town of Flint about their hindsight regarding proper water treatment, shall we?

            On August 20, 2020, the victims of the water crisis were awarded a combined settlement of $600 million, with 80% going to the families of children affected by the crisis. By November, the settlement grew to $641 million. In January 2021, Snyder and eight other officials were charged with 34 felony counts and 7 misdemeanors—41 counts in all—for their role in the crisis. Two officials were charged with involuntary manslaughter.

        • by cusco ( 717999 )

          the county or city should be able to afford it and share that resource across departments

          This is Flori-DUH, that's not likely to happen. Additionally Oldsmar is a suburb of Tampa/St. Petersburg, where brains go to die.

        • > doesn't seem to have that excuse. Even if the water treatment plant can't afford a good IT security person to help full time

          They *could* have got me for a day to tell them what's wrong, a day to help with questions about exactly how to solve it, and later a day to sign off on the new systems.

        • Using good firewall rules, data diodes, and physical separation requires more Ethernet cables and switches! And it requires somebody to figure out how to burn updates to a disc and apply them that way! Do you think the operations team is made of money, huh?

          Rather ironic I'm driving many miles today to perform DVD-driven updates on standalone systems. Perhaps they're not "made of money", but should be wise enough to understand risk. Needless to say, a company isn't going to find much sympathy from me when the billion-dollar lawsuits fly after a hacker kills half a damn town poisoning the water supply because you didn't want to spend $20K on one-time infrastructure costs and hire a $50/hour part-time contract worker to do the onsite maintenance.

          Hell, this is

          • by Entrope ( 68843 )

            Your reaction seems a little overblown, considering that the fallback security control -- an on-site human operator in the loop -- quickly detected and corrected the attack. Human lives were never at risk.

            Maybe a more capable attacker would have been able to compromise that on-site operator's status displays, so that they did not show the elevated levels of dangerous chemicals. Maybe there are other security or safety measures that protect against that kind of thing. They should still remedy the security

            • Water samples are continuously tested and any dangerous condition would have been noticed before the water was allowed into the system. The public was never in any danger
      • by taustin ( 171655 )

        It isn't hard to understand why it happens, though. The cost savings of having contractors have remote access are a real lure to the people who make the decisions, who are generally money people, not technical people.

        That's how Target got hit, after all.

        (This was apparently though TeamViewer?)

        • by sjames ( 1099 )

          IIRC, the Target hack came in through an HVAC vendor. It could have been prevented by isolating the HVAC from the POS (can't imagine why a cash register needs to talk to the thermostat).

          It didn't even need to be physical isolation, properly configured VLANs could have prevented the hack.

      • by rjune ( 123157 )

        You are absolutely correct on this: "Yeah, or we "smart" adults could maybe realize these kinds of systems should not be remotely accessible." I have worked in a water filtration plant (as a chemist, not an operator) and was impressed at the level of monitoring being done by the operator. Many times, automated systems produce massive complacency. In another surprise, what Gualtieri said was correct. An overfeed for that short of a time would be quickly diluted to safe levels.

        I

    • So...exactly WHY is a system like this not air gapped at least from the general internet?

      What reason would this system have to be capable of being accessed by the internet?

      • Re: (Score:2, Insightful)

        by Entrope ( 68843 )

        TFA says the remote access capability was regularly used so people can diagnose problems remotely. That makes some sense, although one would think that remote users should be restricted to equipment that only has read-only access to the system. But maybe their control systems do not support that kind of separation, so it would require extra equipment to add that security protection.

        • by rgmoore ( 133276 )

          When troubleshooting, it's frequently helpful to be able to change things rather than just look at them. You could depend on having an on-site user interacting with the troubleshooter to do the hands-on changes, but that's still cumbersome.

          • by bjwest ( 14070 )

            When troubleshooting, it's frequently helpful to be able to change things rather than just look at them. You could depend on having an on-site user interacting with the troubleshooter to do the hands-on changes, but that's still cumbersome.

            How about keeping competent personnel on hand at critical infrastructure sites capable of handling problems on site without exposing the system to hackers?

          • by jbengt ( 874751 ) on Monday February 08, 2021 @07:25PM (#61042168)

            When troubleshooting, it's frequently helpful to be able to change things rather than just look at them.

            If 100 ppm is normal and 11,100 ppm is toxic, it should not be possible to set the system to 11,100 ppm.
            Even if were to turn out necessary to set the lye level up to toxic levels for cleaning or some other relatively rare reason, it would not be necessary to do that remotely.

            • In an oversimplified world where people without knowledge make assumptions, such might be the case.
            • by Cederic ( 9623 )

              Ah, but what if you could override the system tolerances? Or changing the water flow so that the low levels of additives get added multiple times? Or adjust the inventory records to fool the system into thinking the additives are 0.1% of their actual toxicity?

              Even with the "don't let people do that" option you're suggesting, once the system is compromised all bets are off.

              I do though accept that a simple unauthorised account access is still restricted to default behaviours for that account, so I would still

        • It will be interesting to see where the 'hack' originated. My guess is that someone enabled some easy remote access function due to Covid remote work, and one of the remote workers got exploited, or maybe even one of his/her kids got on the thing. They probably need to beef up their authentication via an intermediate system.
          • I heard SolarWinds makes a good system for that.

            On second thought, no... disconnect it from the internet permanently. Regardless of Covid, the Black Death, aliens, or anything else. The water supply is one of the few things that needs to be kept secure and running in these events. For perspective, here in TX we have strip clubs that stayed open at full capacity because they claimed to be "restaurants"... I don't imagine Florida did any better.

      • by Darinbob ( 1142669 ) on Monday February 08, 2021 @05:48PM (#61041840)

        1) People want to work remotely. Not necessarily at home, but in the office and not in the actual water processing plant. Especially if you have 100 controls which are at 100 different sites and would otherwise have to pay someone to drive to them all regularly. Money gets saved. And money is the biggest motivator.

        2) Even computer oriented companies have great difficulty hiring exceptional security experts. Why would a water plant, factory, or other industrial site do better at computer security? After all, everyone connected dutifully to the cloud just like Microsoft, Amazon, and Google told them to. What can't be handled in the cloud they will outsource to someone else. Money gets saved!

        3) It's like asking why the water company hired Barney Fife as a security guard instead of getting professional war-trained mercenaries to guard things. I mean the citizens would revolt if they found out their fees were being spent on expensive guards, expensive computers, expensive consultants, expensive workers, etc.

        • 2) this is an easy to fix problem: train security engineers

          When did training your employees become everyone else's responsibility? There's no labor shortage; there's a competent management shortage.

          • When did training your employees become everyone else's responsibility?

            Since some are convinced that one should do things for the love and not the money. Hereafter artists and open-sourcers took this to heart working and learning on their dime, giving their efforts to "everyone else"'s .

          • by Cederic ( 9623 )

            Boss: "We need remote access to that facility for operational and cost control reasons."

            Diligent employee: "It's going to be a security risk. Can we train up some security people?"

            Boss: "Nah, we're a fucking water company. We know shit about this. Hire in an external to get this set up for us."

            The company has thus done absolutely nothing wrong by not training their own security staff.

            The fact that the issue was spotted and addressed actually demonstrates that they had very good operational procedures (or an

        • 0) 24/7 monitoring was required but 24/7 staffing payroll wasn't provided.

    • by Tablizer ( 95088 )

      [what if] humanity was ultimately destroyed by a bored teenager breaking into...

      More or less the plot of War Games.

    • If it was a 'bored teenager', and that 'bored teenager' lives in the U.S., then when that 'bored teenager' gets out of Federal prison, he won't be a 'bored teenager' anymore, he'll be a 'bored 40- or 50-something' more than likely -- and have 'domestic terrorist' on his permanent record.
    • by Cylix ( 55374 )

      I am reminded of this quote from hackers...
      "What are you, stoned or stupid? You don't hack a bank across state lines from your house, you'll get nailed by the FBI. Where are your brains, in your ass? Don't you know anything?"

  • I have an idea (Score:3, Insightful)

    by slashmydots ( 2189826 ) on Monday February 08, 2021 @05:23PM (#61041744)
    Maybe don't put the system that controls that on a network?
    • by cusco ( 717999 )

      So the next time that system is out of variance it has no way of throwing an alarm? You might be surprised by how much infrastructure equipment is in isolated buildings in the middle of nowhere that actual humans visit once in a blue moon. There is a control building for the water system in the village where we have our cottage, I've noticed a blackberry vine grown across the doorway for the last year so no one has actually been in the building that long. I've worked at an electrical utility where they m

      • by AmiMoJo ( 196126 )

        If all you need is monitoring then point a webcam at it. Seen that done before.

      • Blackberries grow on a cane, which has wicked thorns that can rip your skin to shreds.

        That is if what you see is indeed a blackberry plant and not the vine of something else. In that case, the building is much less secure.

        • by cusco ( 717999 )

          We live in the Pacific Northwest, we're much better acquainted with blackberries than anyone really wants to be. I paid my rent one month hacking the blackberry canes that had taken over the landlord's property (got really, really good with a machete).

        • by Whibla ( 210729 )

          Blackberries grow on a cane, which has wicked thorns that can rip your skin to shreds.

          When being pedantic about terminology it would seem sensible to use the correct terminology throughout your statement:

          "First- and second-year shoots usually have numerous short-curved, very sharp prickles that are often erroneously called thorns."

  • Aggressive water in a region where they are still having lead pipes could be very dangerous.
    But luckily this is Florida, where the water is disgusting anyway and people don't even use it to make ice, much less drink it.

    It's that free water on the table that every customer was entitled to, dear kids, in the olden times, when we still met in places called 'Restaurants', I remember it fondly.

  • by oldgraybeard ( 2939809 ) on Monday February 08, 2021 @05:38PM (#61041790)
    to the public internet How? Why? Sounds like failure by design to me. Who is in charge? Are they still in charge?
    • by Tablizer ( 95088 )

      IOCT: Internet of Compromised Things. My kids will face a strange strange future (besides their father).

      • A strange future like it was 20 years ago, when everything wasn't "connected" and things went along fine.

    • I’m sure they will fire the guy who sweeps the floor.

    • by AmiMoJo ( 196126 )

      Would being disconnected be any better?

      They aren't going to pay someone to monitor it 24/7 so disconnected just means more vulnerable to undetected faults and physical tampering.

      It might be perfectly secure, the hack could be an inside job or someone in their main network.

      • by Lyance ( 7545382 )
        Yes but realistically speaking the threat is likely to come from foreign state actors. Plus, given these are low security municipal systems an inside man is going to be able to wreak havoc regardless of what computer security is implemented. It's not like every room is going to be individually access controlled. One knowledgeable guy set loose in an industrial facility for an evening could *easily* put the facility out of commission for weeks or months without ever touching a computer.
      • by Lyance ( 7545382 )
        One more thing I forgot to mention. It's entirely possible to get the best of both worlds. Create an internet connected system for process monitoring, and a separate system for control. There's no reason an operator can't walk over to an isolated panel to punch in the hydroxide additive rate. This is simple, and it has the added benefit that both systems need to be compromised for the system to be covertly sabotaged.
      • Would being disconnected be any better?

        Yes

        They aren't going to pay someone to monitor it 24/7

        It's the city water grid so why not? The building is going to be occupied and running 24/7 so should the operator's seat.

        so disconnected just means more vulnerable to undetected faults and physical tampering.

        It might be perfectly secure, the hack could be an inside job or someone in their main network.

        The controls shouldn't be accessible from their main network either, ideally the whole control system is air-gapped.

        It doesn't mean 100% secure, you can still have disgruntled employees or some hacker team from a Hollywood movie showing up, but it makes attacks much harder. [mcafee.com]

        But if you're online you're basically exposed to the entire Internet. Even to malicious hackers who are smart enou

        • The whole thing is silly anyway. Anything "important" like this should have multiple levels of redundant checks from different, disconnected systems. You raise level to 11,100? Oops, rejected - value seems weird. If you don't like it, go to this other system completely firewalled and change the rules. OK, you did that? Great. Uh oh, the downstream check failed - value of of bounds error. Better send someone down to the basement to force and allow a manual override.

          Whew, OK I got my 11,100 level in. Oh. Shit

          • The whole thing is silly anyway. Anything "important" like this should have multiple levels of redundant checks from different, disconnected systems. You raise level to 11,100? Oops, rejected - value seems weird. If you don't like it, go to this other system completely firewalled and change the rules. OK, you did that? Great. Uh oh, the downstream check failed - value of of bounds error. Better send someone down to the basement to force and allow a manual override.

            Whew, OK I got my 11,100 level in. Oh. Shit. Now the pre-dispensing system is complaining. Someone will have to go and push some buttons to allow this unusual condition. Send John. OK, done! Well, except now the dispensing system is hard-limited to 300. Shit, now we have to go in with a fucking special screwdriver after we've unlocked the chassis and make 10 adjustments.

            My guess is that while it's probably not this hard (it should be), in reality that hacker didn't get as close as he thought. And now he's going to reap the whirlwind, they won't fuck around in trying to find them.

            The article isn't clear about how the "11000" was set. Was it a target value written directly to a PLC that would start injecting chemical until it hit the target value or was it a value the SCADA system itself used to manage the injection? (no reason to assume the device that adds the chemical is also the device that can measure the levels).

            The problem with all those checks you propose is that these systems are both heavily customized yet still need to be ridiculously reliable (don't want a bug writing 111

        • by AmiMoJo ( 196126 )

          They won't pay someone to monitor it because you won't stand for that added cost on your bill/taxes.

          Air gaps won't stop someone controlling the system. If you put a minimum wage operator there they will do what they are told via spoof email or proof phone call. It's probably even more vulnerable, now they don't even need to hack your network, just get an email address or phone number.

    • So that the employees monitoring/controlling it can work from home.
  • If this is a foreign gov't in action, it can be considered an act of war, and a war crime.

    • No one except government (foreign, domestic, whatever) would connect a utility to the "Bat Shit Crazy Show" which is the public internet at all.
      • by Tablizer ( 95088 )

        No, I mean an enemy nation hacking into US infrastructure and screwing with it. Yes, the builders connecting it to the public internet is dumb, but two wrongs don't make a right.

        Forgetting to lock your front door is not a sufficient reason to dismiss your murder by a robber.

        • by cusco ( 717999 )

          Does no one know what "remote in" means any more? Holy carp. This used to be a techie site, what the frack happened?

          They didn't connect the water plant directly to the Internet, some things are too stupid even for Flori-DUH. Someone connected to their network, almost certainly over a VPN connection. Once they had done that they have access to everything on that LAN. I'd be very surprised if this doesn't turn out to be someone's kid who got playing around when their parent forgot to log out, saw an inte

          • by Tablizer ( 95088 )

            Remoting in over the internet is still "connecting to the internet". Since land lines are dying, almost any remote service of any kind pretty much is forced to use the internet. (And land lines were hackable also, ask Woz.) Maybe there are ways to completely avoid anything touching the internet, but it would likely be expensive or inconvenient.

            • by cusco ( 717999 )

              Power system SCADA networks are supposed to be air gapped from the rest of the great wide world, including corporate LANs, and almost all of them are. And you're right, it's expensive and inconvenient but the risk coming from NOT doing that is to large.

  • by whoever57 ( 658626 ) on Monday February 08, 2021 @05:45PM (#61041828) Journal

    At least the drains would have been clean and free of obstructions.

    '"Sodium hydroxide, also known as lye, is the main ingredient in liquid drain cleaners," Gualtieri added.'

  • My guess is? (Score:2, Insightful)

    by King_TJ ( 85913 )

    I bet the system that was breached was just a Windows PC running remote control software like TeamViewer.

    The way packages like that work, there's usually at least one paid account/login that's owned by whoever in I.T. does the remote support, and once you sign into it - the app gives you a list of all of the workstations that are configured for one-click remote unattended access.

    Obviously, it CAN be configured in much more secure ways. But this is yet another situation where ease of use/administration plays

  • "Hacker"? (Score:2, Interesting)

    by Pascoea ( 968200 )
    Was this a "hacker" or a script kiddie that discovered an open RDP port and guessed the default password of "123456"? Or was it an operator that wasn't paying attention while trying to change the value from 100 to 110, then covering his ass by saying "it must have been a haxxor"?
    • If it was RDP, the operator would have been logged out (even if their same credential was used) as soon as the perpetrator logged in. I'm thinking something more like TeamViewer with a weak password.
  • by hamburger lady ( 218108 ) on Monday February 08, 2021 @05:54PM (#61041860)

    why should i believe the water treatment plant? these guys literally lye for a living.

  • Not a hack (Score:5, Interesting)

    by reanjr ( 588767 ) on Monday February 08, 2021 @06:02PM (#61041890) Homepage

    I'm betting there is no hack. Just a fat-fingered operator. You don't change 100 to 11,100 as a hack. That looks much more likely to be a key repeat issue.

    • Fat-fingered hacker?

      But your explanation is much more plausible and accounts for a common human behavior - lie about the lye. Your interpretation will be the correct one.

      • It's intereting how people like you are willingly black-eyed and always choose the most comforting and harmless possibility, regardless of having amy knowledga making any of the choices more likely.

        Like conspiracy theorists always choose the most out there one.
        You both just pick what reaffirms what you want to be real. It's just that they need the crazy fantasy to explain that they can't handle the world, and you need the comforting fantasy because you can't handle the crazy of the real world.

        I'm sorry, Occ

        • It's intereting how people like you are willingly black-eyed and always choose the most comforting and harmless possibility, regardless of having amy knowledga making any of the choices more likely.

          Like conspiracy theorists always choose the most out there one.
          You both just pick what reaffirms what you want to be real. It's just that they need the crazy fantasy to explain that they can't handle the world, and you need the comforting fantasy because you can't handle the crazy of the real world.

          People are not gods. Everyone as a matter of necessity has to make decisions based upon incomplete information. There is nothing wrong with having an opinion based on little to no evidence. It is when you take that extra step of disregarding the fact the opinion you formed has a tenuous evidentiary basis that you get into trouble.

          Lets consider your own words "people like you are willingly black-eyed and always choose the most comforting and harmless possibility" followed by derisive commentary. Here you

        • I'm sorry, Occam's razor is a logical fallacy. At least in its regular interpretation.

          While there is some truth to what you say, it's a non sequitur. No one invoked Occam's razor. They just used subjective probability and picked what they think is the most likely/plausible cause. You could argue against their assessment, but I'm cynical enough to think there is a decent chance they are right.

    • Stuck key more likely. Ever seen computer "type by itself" when a key gets stuck? Spooookey...

    • Why wouldn't a lazy hacker just press a single key repeatedly?
    • Hackers always do funny numbers like 69,420
  • We've been told it's unhackable [youtube.com]
    .

  • Wouldn't (shouldn't) there be sanity checks on changes like that ?
  • 11100 ppm means 11.1 mg of sodium hydroxyde for 1g of water. Molar mass of sodium hydroxyde is 42 g/mol, we have 264 mmol/L. pH would be 10.42. Did I get it right?
  • *crosses fingers*

    *drinks pire medicinal alcohol and rain water*

    *starts reading TFS*

  • Clever girl.
  • We changed the setting back and kicked the intruder... mission accomplished...

  • "adjusted the amount of sodium hydroxide in the water from 100 parts per million to 11,100": afaik, he did NOT. He apparently changed the set point to that, but it seems improbable that the system could have physically achieved that set point, and probably couldn't even come close. It would have taken extreme over-engineering of valves, pumps etc. to be able to put that much NaOH into a system when it was supposed to be two orders of magnitude lower.

Help fight continental drift.

Working...