Now Meltdown Patches Are Making Industrial Control Systems Lurch (theregister.co.uk) 98
Patches for the Meltdown vulnerability are causing stability issues in industrial control systems. From a report: SCADA vendor Wonderware admitted that Redmond's Meltdown patch made its Historian product wobble. "Microsoft update KB4056896 (or parallel patches for other Operating System) causes instability for Wonderware Historian and the inability to access DA/OI Servers through the SMC," an advisory on Wonderware's support site explains. Rockwell Automation revealed that the same patch had caused issues with Studio 5000, FactoryTalk View SE, and RSLinx Classic (a widely used product in the manufacturing sector). "In fairness [this] may be RPC [Remote Procedure Call] change related," said cybersecurity vulnerability manager Kevin Beaumont.
Industrial systems should be super-simple (Score:5, Insightful)
In general, simpler systems have a smaller attack footprint.
Like the rest of the computer industry, many industrial systems are more complicated than they need to be.
Yes, industrial equipment is simpler-by-design than your average general-purpose computer, but there are still some "because we can have it and it would be a nice thing to have, we have it" or "because we can buy an off-the-shelf chip that does things we don't need cheaper than paying the chip-vendor to disable unneeded functionality, we do" situations.
There are probably innumerable industrial-control systems that can run their core functions "intelligence" on the equivalent of an early-1970s microprocessor or less. Perhaps they should.
Re: (Score:1)
No, they should just use standard Ubuntu/Debian/CentOS or something because then at least you can update the thing, and just implement their own thing in python so that you can easily read the code and inspect what it is doing.
Re: Industrial systems should be super-simple (Score:4, Funny)
VB6 and .NET coders are cheap and the code can be cobbled together (although unreadable) with the most guaranteed billable hours and most expensive support packages ever.
Re:Industrial systems should be super-simple (Score:5, Insightful)
Yep, you as a control system owner can either buy (a) what's behind door number 1 that does everything you'd ever want and walks the dog when you are too tired, all for the low, low price enabled by the manufacturer selling millions, (2) what's behind door number 2 that does precisely what you want because you specified and contracted that system for your operation, all for the high, high price forced because you require a one-off.
By the way, what's behind door number 1 comes with a volume discount so you can use it in several places in you operation. What's behind door number 2 comes with a volume discount of one because its a one-off.
Choose wisely.
Re: (Score:2)
You forgot doors 3 and above:
Door 3:
Actual 1970s-era chips or slight revisions of them that are still being produced: Why do I need a system with an 80486 or modern-ARM-chip with a wired or wireless Ethernet Ethernet interface when a 4- or 8-bit microcontroller and a simple wired or wireless serial interface will do?
Door 4, more expensive, feasible only if you are ordering tens or hundreds of thousands:
Same microcontroller and same serial interface controller but with features you don't need removed during
No they shouldn't, simplicity is too complicated (Score:2)
an 8-bit micro-controller won't cut it anymore.
Everything is so precision motion-based and customizable.
Everything needs to have comprehensive diagnostics to say what's wrong (because troubleshooting has been replaced with magical thinking).
Everything is running at multiples of 125 microseconds, now even update rates of 10 milliseconds is too long.
Serial (RS232) is an absolute nightmare to diagnose. RS485 is a nightmare to get them to wire it correctly. YES, RS485 as in 2 wires and a shield is still an abso
Re: (Score:2)
Spot-on. Not sure if you are being sarcastic on a couple of the points, but when you have access to second and third or fourth derivative data rather than a simple counter you have much more valuable diagnostic data to work with to streamline a process. It might not be needed continuously, but to have it at all means you get it continuously.
Yes, it can make people lazy-- we will just tune it in the field-- but for some things a 1% delta is really important.
Re: (Score:2)
So go with a 32 bit microcontroller with an ethernet port.
Re: (Score:2)
This has always been an option, and the industry chose Door #1 years ago because it is by far the most productive and economical.
It's not like the machinery is getting any simpler either. How much data do they send and how often do they need to report? Do they integrate with inventory systems to track usage of raw materials and other consumables? How good are the predictive wear/failure alerts?
More precision and more complex automation are going to push those 1970s-era control systems from outdated to unwor
Re: (Score:2)
This has always been an option, and the industry chose Door #1 years ago because it is by far the most productive and economical.
Three competing companies are on the same street; one chose door #1, the others chose door #2. One of the ones who chose door #2 has higher productivity and TCO than #1, because the engineering consultant they hired delivered what they promised. The other third company is about to be shut down and liquidated because their consultant never delivered, and after throwing good money after bad, they eventually opened door #1 but because of their debts they didn't have the cash flow to buy large enough quantities
Re: (Score:2)
Door 5 is cheaper, is clearly used for a similar purpose by companies larger than yours, but the manual in only in Chinese and the marketing documents in "English" don't make any sense.
Re: (Score:2)
Those '70s era microcontrollers easily fit on a cheap FPGA, allowing full customization.
Re: (Score:2)
As a software consultant I can inform you, you understood door 1 just fine. But you didn't really understand door 2 very well. It does actually scale, you just fell off your knowledge cliff. When you used the word "specified," that includes scaling.
Re: (Score:2)
There's a whole high volume world of microcontrollers out there that are more powerful than the chips of the '70s and available dirt cheap. There'd be even more in use but programmers who can write software for them are slightly more expensive.
It's not just the CPU. The microcontrollers tend to have very simple requirements for support on the board. Some can even be trivially plugged into a breadboard and brought up with 5 or so wires.
Re: (Score:2)
If we can get more developers in the industrial sectors to swap to something more cross-platform then they could use locked down ARM devices like Raspberry Pi's that can be swapped out more easily to check for security audits.
Re: (Score:1)
The systems affected are not the actual PLCs that directly control the equipment.
These are the HMIs and programming software that runs on a windows workstation.
(The Panel-Mounted HMIs also run typically WinCE, although the newer PV5500s run linux)
And those are *anything but simple*.
I run all my industrial programming software in VMs for just that reason--there are bizarre dependencies and incompatibilities between different manufacturers of this software, and even between different vintages of software from
Re: (Score:2)
The problem is actually not the SCADA system itself. It's the computer monitoring the SCADA equipment!
The meltdown and spectre patches that Microsoft released are apparently causing problems with the monitoring and configuration software that runs on the computers attached to the SCADA network. The equipment is fine.
Re: (Score:3)
A little off on your description. SCADA is Supervisory Controls and Data Acquisition. There are several parts and pieces to that and in most systems, that included the operator interface which is usually run on Windows machines.
Yes, the equipment that interfaces with the field equipment is fine, but the operators can't see what that equipment is doing.
It would be like saying, your car is fine and the engine is running, but your brakes, gas pedal, steering wheel all stopped responding and your windshield i
Re:Industrial systems should be super-simple (Score:5, Informative)
I am a controls engineer and use the software mentioned in this post.
First, controls guys who know anything and don't get IT telling them, you must do this now, will never install a patch until vetted by the manufacturer. I actually got a notice from the vendor saying, don't install this patch 2 days after the patch was available.
As to being more complex then they should be or simple...
The actual controllers that run the process are extremely simple, extremely hardened and designed to run 24/7/365. PLC processors cost $4000-$15,000 depending on type and memory and they get into the hundred of meg of memory.
Where it gets difficult is when you start using PCs to run your operator interface. There are tons of graphics, reports, trends, etc and you use software that is designed to run on Windows, which most of your operator interfaces are designed to do.
When a patch like this hits, the operator interface or historian has issues, but the PLC running the process keeps doing it's job, you just can't see into the PLC.
So yes and no. There are things that are more complex and that could be simplified/run separate from windows, but those start getting prohibitively expensive and the tiny bit of extra reliability is not needed. Those kinds of systems cost 2-5 times as much and the development of those systems is more expensive because there are even fewer people with experience with it. If I had experience with those systems, I would be making 70% more then I am now and I am making enough that I don't need to complain.
Re: (Score:2)
Re: (Score:2)
If you have a system that simple and unimportant that you can throw a desktop computer at controlling it then there will be a properly industrial hardened PLC in your price range. No seriously some proper PLCs from the vendor mentioned in the post run at a lower cost than any computer you can throw at it. Just configure and provide it with 24V.
But chances are if you're having problems with the performance of a Wonderware Historian you're problem is that you have 5000 analogue instruments on site and you're
Re: (Score:2)
The problem, as Target Corporation learned the hard way, is that control systems and their supervisory PCs are often the attack gateway into corporate financial systems. Yes, I'm sure your entity's control systems are never connected to the Internet and are invulnerable - just as Target's HVAC controllers were standalone, not connected to the Internet, and would never be allowed to communicate with the credit card processing system. Problem is they weren't, they were, and they did. If the control system
Re: (Score:2)
Agreed. Even air gapped, a system still needs to have security patches applied.
It doesn't matter how great your protocols are to limit access, included in your policy has to be patching. There are many things you have to consider when securing a network with life critical things attached to it. When and how you apply those patches is just as important as any other parts of the policy to secure your network.
Are my systems at risk from Meltdown and Spectre because I have not patched? Yes. Due to other la
Re: (Score:2)
Probably at least somewhat true that you don't need much of a computer for many industrial control tasks. And even when not true, do the systems have significant sensitive information that needs to be hidden from other users? Is there any point in patching the software against spectre/meltdown if the patches cause trouble and the malware doesn't represent much of a threat to those systems?
OTOH, getting infrastructure OFF the damn internet as quickly as it can be done seems like a very good idea even if do
Re: (Score:2)
Yes, like you were agreeing to, to run an industrial controls systems you don't need much power, it is just the SCADA (operator interfaces and historian) where you start needing a lot more and it is very difficult to do without getting into the PC and server areas.
As to controls systems being on the internet, yeah, those people are idiots or dealing with stuff that is non proprietary, non life threatening.
At my place, there is not outside logging into the system. They need troubleshooting, I drive in. Oth
Re: (Score:2)
Or they could be isolated and unpatched. An airgap is the best security, always.
Re: (Score:2)
This is an interesting question. Air gapped is the best solution, but not always acceptable.
Some places need to allow remote access to their controls systems for troubleshooting purposes because they have few experts and it is impractical to fly your controls engineers all over the place.
Even more common is the need to get historical and inventory information up to the business network real time so people can make proper decisions. Security best practices talk little about air gapping because almost every
Re: (Score:1)
Even more common is the need to get historical and inventory information up to the business network real time so people can make proper decisions.
This is where a one-way air gap can come in handy:
Have the "secure" side continuously report real-time data 24/7 to a "less secure" device for recording/reporting over a one-way channel. Use VPNs or other controls to give access to the "recording/reporting box" as needed.
If the reporting/reporting box gets compromised, at least it can't directly leapfrog back to the "secure side."
a lot of the manufacturing stuff is stuck in the d (Score:2)
a lot of the manufacturing stuff is stuck in the direct hardware access ideas of dos / win 3.1.
Someone needs to do an DirectX / opengl like layer for this stuff to use.
Reliability above all else (Score:2)
a lot of the manufacturing stuff is stuck in the direct hardware access ideas of dos / win 3.1.
That's not necessarily a bad thing. Manufacturing equipment generally doesn't benefit greatly from having the latest and fanciest do-dads and gee-gaws. It needs to be reliable above all else in most cases. Well made manufacturing equipment mostly should never need to be patched. There are some exceptions but they are rare. You do run into some networked CNC and robotics that needs to be more up to date but the presses that my company runs really just need to go up and down. More complexity or features
up to date to just run on newer boards is one thin (Score:2)
up to date to just run on newer boards is one thing. The last boards to do ISA out side of SBC systems are OLD p4's.
Re: (Score:2)
VMware pulled some of their patches (Score:3)
VMware pulled some of their patches
Note: ESXi patches associated with VMSA-2018-0004 have been pulled down from the online and offline portal.
...
For ESXi hosts that have not yet applied one of the following patches ESXi650-201801402-BG, ESXi600-201801402-BG, or ESXi550-201801401-BG, VMware recommends not doing so at this time. It is recommended to apply the patches listed in VMSA-2018-0002 instead.
...
For servers using the Intel Haswell and Broadwell processors (see Table 1 for the specific list of affected VMware vSphere supported Intel Haswell and Broadwell processors) that have applied ESXi650-201801402-BG, ESXi600-201801402-BG, or ESXi550-201801401-BG VMware recommends the following:
...
VMware is working closely with Intel and the industry to come to a quick resolution of this Intel microcode issue and provide an update to our customers as soon as possible.
Oops!
reference [vmware.com]
Re:VMware pulled some of their patches (Score:4, Insightful)
I guess I should have finished my thought.
It's not just industrial control systems, but hypervisors, and plain old systems too. It sees like this is an object lesson in how speed (in terms of releasing a fix) comes at a cost of performance/quality. I know people were all in a panic once Meltdown and Spectre became public, but this wasn't just fixing a SQL injection vulnerability in Rails or Django. This fundamentally affected the execution of nearly every instruction to go through affected CPUs.
I suspect that the severity and publicity made a more organized roll out with extensive beta testing impossible for just about every vendor that had affected products.
Stuxnet is now crashing (Score:3, Insightful)
Now things like Stuxnet won't be able to infiltrate as easily. WTF are these things doing connected anyway, and if not connected why do they need the patches? And don't get me started on Windows...
Re: (Score:2)
WTF are these things doing connected anyway
So to answer your question from multiple points:
Stuxnet : Stuxnet did not infect a connected system. It was spread by direct attack via USB. ... if you have access to it. This is very sensitive data which some governments will require you to store in realtime ... offsite.
Connected : A wonderware historian serves the sole purpose of storing long term trends of your entire control system. This is very useful data
Now to make a point: On a well setup and secure site the Historian will sit on a different network
Re: (Score:1)
If you're not living by the example of Jesus (The man from the middle east, not the guy who trims the hedges) you're not a Christian.
Ironic (Score:2)
So the meltdown patches are themselves causing meltdowns? Isn't it ironic! (Doncha think?)
Re: (Score:2)
Like rain on your wedding day.
People use windows for industrial automation??? (Score:2)
Re: (Score:2)
Re: (Score:2)
Yes, absolutely, most industrial automation are run on Windows.
No, it isn't that scary. It can be, but if properly implemented with the right security in mind, you can keep the system up and running reliably.
As stated below, the windows machines are used for the operator interfaces and to record information. The things that actually controls the process are different and unaffected by this and the screwy things with windows.
I am a controls engineer, i.e. program, spec, maintain, industrial controls system
Re: (Score:2)
Toldja so... (Score:5, Insightful)
Re: (Score:3)
Re: (Score:2)
Just because it took 20 years to be discovered doesn't mean that it is going to take long for hacking toolkits to include exploits based on the PROOF OF CONCEPT CODE RELEASED BY THE DISCOVERERS.
Still waiting for a PoC that isn't just a rigged video that someone says is a PoC.
Show me a rogue Web page that reads passwords or other private data, and I'll panic.
Re: (Score:3)
Yeah, pretty much everything GerryGilmore said. This knee-jerk reaction to a pretty obscure flaw is way overboard.
I personally don't want my CPU's branch prediction gimped because some other idiot can't keep his web browser away from malicious sites.
The only panic that should be realistic and warranted is big cloud VM providers concerned these attacks could compromise tenants on shared systems. Patch that shit into oblivion as far as I'm concerned, but get your grubby patches off my desktop. I don't want
Re: (Score:1)
Agreed. I've worked with a lot of these systems. And most companies connect them to their internal network because all the PLCs are network ready. The funny thing is most of the controls PCs don't talk to any corporate resources except possibly for authentication. (even then the users typically all log in to only one controller account). I'd bet in 90% of the cases an isolated network switch and a few patch cables solves their problems.
Also Beckhoff TwinCAT 3 (Score:5, Interesting)
Re: Also Beckhoff TwinCAT 3 (Score:1)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Quote, "I'm OK with delayed patch installation and extra security measures, but every patch needs to be tested by them and certified for installation. They have no mechanism for doing that at all."
They actually do have a mechanism for testing and releasing what patches are acceptable on their systems. This articles talks about Wonderware and Rockwell systems, both of which I use on a daily basis as an end user. Both make available a list of what patches they have tested and vetted out for their systems.
Industrial systems don't have as much spare room (Score:5, Insightful)
Re: Industrial systems don't have as much spare ro (Score:1)
Re: (Score:2)
Re: (Score:2)
I could easily see how billions of dollars worth of industrial systems simply will not be able to patched due performance cost of the patches.
Doubt it. The systems themselves are unaffected by this mess and don't need patching in any case (controlled access and all). What is being patched here is a handful of database servers. Spend the $3000 to buy a better one. In the grand scheme of "industrial control" not only will it not break the bank but you can most likely pay for it using petty cash.
Maayyybe (Score:2)
If your control systems weren't connected to the Interwebz so you could save a little money on payroll, you wouldn't have to apply the patch.
Re: (Score:2)
Yeah, not so much.
Controls systems not connected to the internet still need to be patched and maintained because there are vectors of attack that can still get across an air gap.
Yes, patching isn't as important, but you still have to patch for security and just to be able to stay compliant with software revisions of the software you are using.
FYI, I am a controls engineer, that means I do this for a living. I use the software mentioned that this crashes, but it didn't hit me because I would never apply a p
Re: (Score:2)
Deploy (Score:2)
User:*Computer install patch and do regression test.
Computer: The company has now regressed
The hazards of monoculture (Score:4, Insightful)
relying on one piece of tech is as bad as relying on one food crop.
control systems need Linux (Score:2)
If you're running an important control system using Windows, you're doing it wrong.
Re: (Score:2)
All the more reason (Score:2)
Now you tell me (Score:2)