'Disabling Cyberattacks' Are Hitting Critical US Water Systems, White House Warns (arstechnica.com) 77
An anonymous reader quotes a report from Ars Technica: The Biden administration on Tuesday warned the nation's governors that drinking water and wastewater utilities in their states are facing "disabling cyberattacks" by hostile foreign nations that are targeting mission-critical plant operations. "Disabling cyberattacks are striking water and wastewater systems throughout the United States," Jake Sullivan, assistant to the President for National Security Affairs, and Michael S. Regan, administrator of the Environmental Protection Agency, wrote in a letter. "These attacks have the potential to disrupt the critical lifeline of clean and safe drinking water, as well as impose significant costs on affected communities." [...]
"Drinking water and wastewater systems are an attractive target for cyberattacks because they are a lifeline critical infrastructure sector but often lack the resources and technical capacity to adopt rigorous cybersecurity practices," Sullivan and Regan wrote in Tuesday's letter. They went on to urge all water facilities to follow basic security measures such as resetting default passwords and keeping software updated. They linked to this list of additional actions, published by CISA and guidance and tools jointly provided by CISA and the EPA. They went on to provide a list of cybersecurity resources available from private sector companies.
The letter extended an invitation for secretaries of each state's governor to attend a meeting to discuss better securing the water sector's critical infrastructure. It also announced that the EPA is forming a Water Sector Cybersecurity Task Force to identify vulnerabilities in water systems. The virtual meeting will take place on Thursday. "EPA and NSC take these threats very seriously and will continue to partner with state environmental, health, and homeland security leaders to address the pervasive and challenging risk of cyberattacks on water systems," Regan said in a separate statement.
"Drinking water and wastewater systems are an attractive target for cyberattacks because they are a lifeline critical infrastructure sector but often lack the resources and technical capacity to adopt rigorous cybersecurity practices," Sullivan and Regan wrote in Tuesday's letter. They went on to urge all water facilities to follow basic security measures such as resetting default passwords and keeping software updated. They linked to this list of additional actions, published by CISA and guidance and tools jointly provided by CISA and the EPA. They went on to provide a list of cybersecurity resources available from private sector companies.
The letter extended an invitation for secretaries of each state's governor to attend a meeting to discuss better securing the water sector's critical infrastructure. It also announced that the EPA is forming a Water Sector Cybersecurity Task Force to identify vulnerabilities in water systems. The virtual meeting will take place on Thursday. "EPA and NSC take these threats very seriously and will continue to partner with state environmental, health, and homeland security leaders to address the pervasive and challenging risk of cyberattacks on water systems," Regan said in a separate statement.
Cyberattacks (Score:3)
Where the rubber meets the road of adventurous foreign policy. Some dude in his apartment in random location around the world doesn't suddenly decide to disrupt US water systems for the lulz. These are nation-state actors. Why they would ever want to attack the US, I have no idea.
Re: (Score:3)
That's what you're going with? That you want hard evidence of who is attacking key infrastructure in such a way as to prepare for a large scale 0 day attack on the onset of a kinetic war? Like, instead of evaluating the likely perpetrators based on the goals being achieved, like crippling national infrastructure?
Glad you aren't in charge of any defense infrastructure, anywhere.
Re: (Score:2)
It is proving it out. It isn't as if there is a ton of diversity in control systems for this type of stuff.
If you were a nation state, you'd want to know how disruptive it actually is, what the recovery time tends to be and what the incident response looks like before you launched a wide scale attack - especially if that attack was part of another large operation and was to serve as a distraction etc.
As to why you'd even reveal such attacks are possible, that isn't a secret all the serious people involved
Re: (Score:3)
What I will say is that if you're a nation state, you still ask for a ransom. Worst case, you look like a typical scammer/hacker ring and use that as your cover. Best case, they pay and you get some off-the-budget funding too.
Re: (Score:2)
See I don't know.
I am sure if your a DPRK and the ransom is actually meaningful funds for your operation you might. What if you are the PRC though? I am not as sure.
Follow the money is probably the simplest component there is to forensics and attribution and therefore represents the greatest threat to deniability. Ultimately its the deniability that is the insulation against a cost diplomatic incident. Again if you are DPRK and even Russia where you effectively don't have a working diplomatic/trade relation
Re: (Score:2)
These are nation-state actors. Why they would ever want to attack the US, I have no idea.
TFA provides exactly zero examples of any "disabling cyberattacks" on our water actually happening.
This isn't something that is real. It is just fearmongering.
Re: (Score:3)
Re: (Score:1, Insightful)
Re: (Score:3)
How about three [cnn.com] and four [cnn.com] months ago? Close enough for you?
Re: (Score:1)
Re: (Score:2)
That's not accurate. They link to this article [arstechnica.com], calling out two specific examples, early in the article.
Re: (Score:2)
This time. Quoting the article "...that still used a publicly known default administrator password." That doesn't inspire confidence that the mitigating controls are effective.
Re: (Score:2)
> These are nation-state actors. Why they would ever want to attack the US, I have no idea.
There are at least 4 countries who hate US's guts.
Re: (Score:2)
We had better be doing the same, it's our only hope.
Re: (Score:2)
Some dude in his apartment in random location around the world doesn't suddenly decide to disrupt US water systems for the lulz.
You couldn't be more wrong. Finding an opening ANYWHERE will allow the curious mind to delve into what was discovered. The fact that it is a water treatment plant of some sort is irrelevant. Making things happen is fun, especially when it gets a large reaction out of people.
All of that being said, I would imagine most of the hacks are indeed hostile foreign State actors... but your assumption that it is NOT State actors is clearly wrong.
Why? (Score:5, Insightful)
Why the hell is all this stuff online again?
Re: (Score:3)
Re:Why? (Score:5, Interesting)
You really don't get it? It's this ignorance and passivity that's the problem. I don't know who you are or what your job title is, but I'm an engineer, and have done some control system / PLC stuff. I have NEVER gotten to make serious decisions. Up the chain the money-oriented business-types make the decisions. Money is always the deciding factor. Often kewlness pushes things too. Oooo look at how I can view the control system from afar. Oh look how I can eliminate an onsite technician's job and save MONEY.
Among many other things I'm admin for a small hosting operation. For years the attacks (pw guessing on ssh, ftp, etc.) were fairly consistent. Then for a couple of months, January to recent week, they tapered down. Suddenly in the past week or so the attacks have quadrupled. And that's with aggressive IP blockers in place and working. I knew something was up. But where to report it???
Like it or not, ignore it or not, we (USA / West) are at war. We need to ramp up vigilance. No more trying to psychoanalyze other people and countries. Stop believing they're so good and want our way of life. Just like Putin and Ukraine, there are those who HATE us and want to conquer us. Stop thinking that modernization and education will change those imperial attitudes. We (the West) have very foolishly empowered China, who aim for world domination. Everyone get your heads out of the sand, open your eyes and ears. China, Russia, Iran, N. Korea, several others, have roots that go back thousands of years. They have deep-seated national and ethnic pride that you can't understand unless you're in and of that ilk. We in the West are largely nomads who've settled down for a bit. We inherently don't understand. But we must observe, learn, and accept the facts and deal with reality or we will be assimilated, and I don't mean that in jest. It's like gravity, or magnetism: we don't truly understand the mechanism, but we can observe it. Get careless and gravity will kill us.
I hope this thing will be a wake-up call, but I fear we'll ride through it and come out even more complacent and naively emboldened.
3rd party vendors don't want local only systems th (Score:3)
3rd party vendors don't want local only systems then need there VPN to talk to there outside service.
Re:Why? (Score:4)
To be fair, even big businesses shouldn't be in charge of national cyber defense. While they are vulnerable to everyone, the attacks aren't coming from just anywhere.
The problem is that our own love of freedom and privacy has been weaponized against us. How do you stop a DDoS that's using a domestic botnet? A Great Firewall won't do anything. Under the PATRIOT act, the NSA was tapping our Internet connections. This is exactly how you would position equipment for national cyber defense. But how do you accomplish that (or get anyone on board) while promising to protect freedom and privacy?
China's Great Firewall isn't any better at defense either, I don't think.
The NSA is officially billed as a combat support agency. I don't know how we have a Space Force but haven't bothered to set up an active branch for cyber defense. And I don't think the US Army Cyber Command should be promoted to that role either. They have a download link for McAfee Antivirus on their home page (no joke).
Re: (Score:3, Insightful)
Nobody wants to conquer the US. But everyone would be really happy if they got the US out of their business. The US has been at war with most of it's existence. What's new is that peak US seems to be in the rear view mirror now, and this has given the rest of the world enough relative strength to manage to fight back. The hate towards US is real, but the US does not need to go about sustaining it, like e.g. the forewer wars main loop. Bomb some brown people - you turn some of them against you - they will at
Re: (Score:1, Insightful)
In general, other nations are neither bad nor good, they're just different, and have interests that differ from the US interests.
In general, you're correct. But there are some nations who are actually bad, not just bad in the society of nations, but bad toward their own citizens. Some of these are week on the international front. Some are strong enough on the international front to be dangerous.
Re: (Score:2)
everyone would be really happy if they got the US out of their business
Right until China or Russia showed up.
Ask Ukraine how happy they would be to get the US out of their business.
Re: (Score:2, Flamebait)
Or lack of responsibility + punishments? (Score:2)
I get it, you're in a bad spot fighting something you can't win against. But I read inside your answer that "powerful people mandate it must be so, and don't understand why it's a bad idea or need to pay the effort to deal with it afterwards".
But isn't the root of this, that hosting companies and network providers aren't responsible for what they do to other people/companies? They have customers who act on the parent's behalf (using the provided services)... doing bad things over and over, and it's the r
Re: (Score:2)
Up the chain the money-oriented business-types make the decisions. Money is always the deciding factor. Often kewlness pushes things too. Oooo look at how I can view the control system from afar.
I've met narrow minded engineers like you before, you boil every decision down to money ignoring the many things that require interconnected networks. Basic optimisation not only improves money but also improves safety. When talking about infrastructure, money is often interlinked with reliability. What may look like saving money to you, is ensuring critical infrastructure is adequately monitored to assure its uptime to someone else. And that's before you get into some regulations requiring virtual realtime
Re:Why? (Score:5, Interesting)
Water system employee here. Firefighting is the most urgent reason for having things available remotely. We have people on call 24x7 to activate pumps and reroute water to maintain the water pressure needed for firefighting. We're part of the 911 system and are notified of major fires. An apartment or industrial fire can require large amounts of water be delivered quickly to maintain pressure.
Second, a lot of our flood control gates and pumps are in remote areas. Heavy rains may require that these be operated at odd hours. We also allow our plant operators to WFH regularly. This requires remote access (via VPN) to do their jobs.
We're lucky in that we're a large city and have (almost) sufficient funding. We can afford to have a security team just to keep out the bad guys. Some of the smaller neighboring systems aren't so lucky and they just hope that their low-bidder AV software will be enough.
Re:Why? (Score:4, Insightful)
That explains why your systems have to be networked, but not why they have to be internet-connected.
Re: (Score:3)
well when the local phone cable changes say $100/mo site for an basic internet link vs an DIA at $300-$1000/mo + hardware rent fees for lower speeds down then the basic link.
And direct point to point costs more may have high install fees if the need to run say fiber to each point and force you to have rented ISP hardware on the network that the ISP controls.
Re: (Score:2)
Proper connections may be more expensive, but proper things usually are. Ya git what you pay for.
Re: (Score:2)
but when the Proper connections are $5K-10K/mo (plus big install fees) vs $500-$1K/mo (free install) for the cheaper way what will the people who control the budges do?
Re: (Score:2)
If Xiputin knocked us into the dark ages, you'd be wishing we'd pay that $10k, assuming you are still alive to regret the cheap route.
Maybe if the Fed gov't coordinated standards and technology for non-web infrastructure communications, we could get economies of scale. Each agency and state reinventing the wheel will keep it expensive.
Re: (Score:2)
Having water system operators responding to fire or flood calls from home pretty much demands an internet connection (via VPN). We aren't staffed 24x7, but our ISO insurance rating demands that operators be able to respond to fire calls within a short time frame. Having licensed operators manning an air-gapped console 24x7 would be extraordinarily expensive, plus fire calls of the magnitude requiring an operator to respond just don't occur that frequently.
The critical systems in question aren't directly i
Re: (Score:1)
> We aren't staffed 24x7
Maybe they should be. "Penny wise pound foolish."
Re: (Score:2)
Perhaps we need to go back to the disconnected model that worked in the past that really wasn't "that" long ago really....
Re: (Score:2)
That explains why your systems have to be networked, but not why they have to be internet-connected.
You can't remotely network systems without a layer to the internet at some point. These systems are *not* "internet-connected". They are connected to networks which themselves are connected to the internet. Attacks on such system involve the traversal of networks, often starting with the computers of the very person who is telling you that they have a need to control things remotely.
Why still so manual then? (Score:2)
Are there not methods to detect high demand and reroute or increase pressure? Or see higher than expected water levels and pump it out?
Not trying to get rid of jobs here... Those sound like very understood problems that have simple, known solutions. Ones that happen again and again.
I know there are jokes or claims that people can "automate themselves out of a job" in IT. But this genuinely sounds like a few lines of code and some sensors could do the same adjustments on demand. No person or external in
Re: (Score:2)
It's convenient... and a requirement of some SCADA management software. You may think that's silly but it's very real [inductiveautomation.com]. The problem here is regulation regarding remote vital systems is too relaxed. There are good security measures in place but there is SO MUCH more that could be done.
However, the more generic problem is that security lapses are not treated as liabilities. "But perfect software is impossible", you cry. Bug free software may not be possible (under current practices) but vulnerability free sof
Re: (Score:3)
If the software really does need to be internet-connected... at a minimum they could limit the IP addresses allowed to access a given end point. Sure, you'd have to update the list periodically; that might be tedious but it's not difficult.
There'd still be the social engineering vulnerability, but at least you'd be complicating things for the attackers.
Re: (Score:3)
None of it actually needs to be internet connected. Seriously, none of it at all. Even if your system works collectively on a national scale, there are still secure private networking options that aren't physically connected to the internet in any way.
Honestly, vital systems that are unable to function independently of other systems are fatally flawed. That means they should have redundant power systems that can operate indefinitely without the grid or any kind of network connectivity.
Society has become to
Re: (Score:2)
Because it had a network port on it, so they hooked it up. Maybe they never even knew it had network control capability. I don't expect water departments to even think to contact a city IT department to hook up equipment like this.
Re: (Score:1)
Their unions probably require they use the correct people to hook up equipment.
Re: (Score:2)
And since it's water equipment, I'm guessing it's probably the wrong union and wrong employees anyway.
Re: (Score:2)
They should not be directly connected. There are proper/secure ways to allow things like remote access, but proper/secure things tend to cost more money/investment. More to the point, the people holding the purse strings don't understand what makes systems proper or secure, but rather what they can fit in their budget.
The only real solution to this problem is to require some regulated set of security standards and practices, and to also require Third-Party Cybersecurity Insurance coverage for these public
Re: (Score:3)
The boss wants to be able to login from home and doesn't want any complicated passwords.
Re: (Score:2)
Why the hell is all this stuff online again?
Not online and airgapped are not the same thing. These things aren't "online". In many cases there are multiple levels of networks, DMZs and firewalls separating the control from the internet and in any normal capacity you can't communicate to them directly.
As to why they aren't airgapped: Basic modern requirements. These things aren't run from hamster wheels. They are loaded with complex equipment that needs to be monitored in real time, needs condition monitoring and trending, needs to provide that data t
Unplug (Score:3, Insightful)
Quickly unplug all control systems that are connected to the Internet. If for some incredibly stupid reason the software won't run unless it phones home to mommy, replace it, even if you have to downgrade to older stuff that worked perfectly well.
Theyre doing us a favor (Score:3)
Thank You Putin for (Score:2)
...testing our infrastructure so that it's more likely to stand up to Xi's antics during his invasion of Taiwan.
Putin knows he's not going to be around much longer, visibly ill, and is throwing everything he has at us to get a going-away present.
Re: (Score:2)
Really? There are people still holding onto the multi year old idea that Putin is dying of something? Whatever it is it's certainly taking it's sweet time to kill him isn't it? Also, kind of weird he isn't visibility lining up a successor if he's actually dying.
Re: (Score:1)
> Whatever it is it's certainly taking it's sweet time to kill him isn't it?
Could be. There are a lot of fatal diseases that can be postponed for several years with top-of-the-line care.
Re: (Score:2)
The king is in good health until he drops dead...
rigorous cybersecurity practices (Score:2)
Re:rigorous cybersecurity practices (Score:5, Informative)
A water treatment plant is one thing. Water supply is another. It's getting harder than you might think. Once upon a time, most connections were over radio, if remote enough to not be able to run wire. Radios have spectrum issues and path issues, so aren't ideal in some situations - think lake level monitoring or pumping stations at dams where they are in a depression. They also tend to fail periodically and need a lot of power, relative to the measuring devices. They're also expensive.
Then came cell phones. Cell phones have their own limitations for signal and longevity. Then came internet connections.
This applies on both the supplier and the consumer side. If you're running fiber to the home, put the water meter on the internet so people can check for broken pipes if it's a vacation home and you can save money by not sending meter readers around the neighborhoods.
A little creeping feature here, a little creeping feature there. It all adds up. And it's pretty easy for the benefits to outweigh the costs.
Re: (Score:3)
Most of those 150/450 MHz radio telemetry links were not encrypted and if you understand industrial p
Re: (Score:1)
As others mentioned around here, localities do it to save a buck by using off-the-shelf shit. If fit hits the shan, they just retire and leave town. Penny-wise-pound-foolish.
windows 7 (32bit) with teamviewer is easy to hack (Score:5, Interesting)
windows 7 (32bit) with teamviewer is easy to hack into
https://arstechnica.com/inform... [arstechnica.com]
if only they had the funds to do things right and pay for on site staff / good IT software and not teamview in free mode.
Checks and balances (Score:3)
> if only they had the funds to do things right and pay for on site staff / good IT software
Humans and gov't are shitty longer-term planners. There are only 2 known forces to get them to fly right: A) A public crisis that puts the slack on display for all to get angry over, and B) A system of checks and balances whereby people with differing agendas inspect it regularly. (Ideally we'd want "no agenda", but that's asking too much of humans, such that "differing agendas" is the next best thing.)
"A" is limi
Re: (Score:2)
some stuff needs enterprise subscriptions with hig (Score:2)
some stuff needs enterprise subscriptions with high min users.
Maybe if some stuff in was not at the enterprise level subscriptions or if the enterprise subscriptions did not have an 500 seat min then more system will have things done in an better way.
But if your plant that may only need 5-15 people to have access to the control system then you may look at stuff like team viewer.
Re: (Score:2)
This is where something like an association of water departments could get together and share an enterprise level subscription. They could also coordinate water control efforts and come to each other's mutual aid.....
But then again, this IS the United States we're talking about, and that sort of shared responsibility and accountability is largely derided as "Socialism" or "Big Government Overreach" so it will never happen.
share an enterprise level subscription?? (Score:2)
We will cut you off then do stuff like force all them to have the same base AD domain for all departments.
Or say to use multi AD domains then your need our global enterprise level subscription that starts at 2000-5000 seats.
Re: (Score:2)
I get it...no matter what happens, the corporations have to get their money. The plebes will simply suffer.
SSDD.
Re: (Score:3)
Re: some stuff needs enterprise subscriptions with (Score:2)
This.
But when the PHB asks why we have to use a secure but not free solution. Instead of Team Viewer. For free. Are you going to be the one to tell him that it's good enough for IT support when he can't find his "Any" key? But not for serious work?
Lack of knowledge in the industry (Score:5, Insightful)
I have vendors asking for port forwards all the time also, but rarely do that for them anymore, I will give them our VPN client (which only gives them access to the network they need), or pass http traffic through a "reverse" SSL proxy that requires 2FA (looking at replacing that with Cloudflare zero trust).
I think it's more of the thing where the convergence of OT and IT needs a little more cross-training between roles, than throw money and regulations at it. I started out my career doing industrial automation (PLCs, PID controls, etc) in the early 1990s, and then quickly picked up (secure) networking later in the 1990s as it became apparent that was the future of this stuff. It never occurred to me that any of this stuff should be world route-able. Always put OT on their own network segments from the start, but not all automation techs have a grasp of networking (and many seem to not want to learn), and simple do what they can to "make it work".
Re:Lack of knowledge in the industry (Score:4, Interesting)
Working with a systems integrator on a new SCADA system for the water/electric/wastewater utility I work for a few years ago, I designed system to be all IP connected (using our private fiber along utility lines, and site-site VPNs to locations not on fiber network), they had never really done an IP system before, but was impressed on how well it works when they were done....Got a call from the systems integrator a few months later at another customer site saying something along the lines of "I have the cable company here (at a remote site for their customer), how many static IPs should we be asking for, one for PLC, HMI, etc".....to which my answer was "how many do you need to make your FIPS compliant VPN work?" and I could hear a blank stare at the other end of the phone....glad I caught that one before it showed up on Shodan or something, but how many others do the same thing? SMH
I have vendors asking for port forwards all the time also, but rarely do that for them anymore, I will give them our VPN client (which only gives them access to the network they need), or pass http traffic through a "reverse" SSL proxy that requires 2FA (looking at replacing that with Cloudflare zero trust).
VPN isn't really good enough, either. It's still exposed to the internet, and vulnerable to DDoS that can deny access during critical events, which could cause just as many problems as direct access to systems could.
Moat Money Moat Problems (Score:2)
Airgap everything!
(Watergapping everything should also be fine, if that's more convenient.)
Re: (Score:2)
Airgap everything!
Because that worked so well for Iran. [wikipedia.org]