Follow Slashdot stories on Twitter

 


Forgot your password?
Close
typodupeerror
×
Bug Security IT

Now Meltdown Patches Are Making Industrial Control Systems Lurch (theregister.co.uk) 98

Posted by msmash from the clock-is-ticking dept.
Patches for the Meltdown vulnerability are causing stability issues in industrial control systems. From a report: SCADA vendor Wonderware admitted that Redmond's Meltdown patch made its Historian product wobble. "Microsoft update KB4056896 (or parallel patches for other Operating System) causes instability for Wonderware Historian and the inability to access DA/OI Servers through the SMC," an advisory on Wonderware's support site explains. Rockwell Automation revealed that the same patch had caused issues with Studio 5000, FactoryTalk View SE, and RSLinx Classic (a widely used product in the manufacturing sector). "In fairness [this] may be RPC [Remote Procedure Call] change related," said cybersecurity vulnerability manager Kevin Beaumont.
This discussion has been archived. No new comments can be posted.

Now Meltdown Patches Are Making Industrial Control Systems Lurch More

Now Meltdown Patches Are Making Industrial Control Systems Lurch

Comments Filter:

  • Industrial systems should be super-simple (Score:5, Insightful)

    by davidwr ( 791652 ) on Tuesday January 16, 2018 @01:35PM (#55939683) Homepage Journal

    In general, simpler systems have a smaller attack footprint.

    Like the rest of the computer industry, many industrial systems are more complicated than they need to be.

    Yes, industrial equipment is simpler-by-design than your average general-purpose computer, but there are still some "because we can have it and it would be a nice thing to have, we have it" or "because we can buy an off-the-shelf chip that does things we don't need cheaper than paying the chip-vendor to disable unneeded functionality, we do" situations.

    There are probably innumerable industrial-control systems that can run their core functions "intelligence" on the equivalent of an early-1970s microprocessor or less. Perhaps they should.

    • Re: (Score:1)

      by Anonymous Coward

      No, they should just use standard Ubuntu/Debian/CentOS or something because then at least you can update the thing, and just implement their own thing in python so that you can easily read the code and inspect what it is doing.

    • Re:Industrial systems should be super-simple (Score:5, Insightful)

      by gtall ( 79522 ) on Tuesday January 16, 2018 @01:57PM (#55939883)

      Yep, you as a control system owner can either buy (a) what's behind door number 1 that does everything you'd ever want and walks the dog when you are too tired, all for the low, low price enabled by the manufacturer selling millions, (2) what's behind door number 2 that does precisely what you want because you specified and contracted that system for your operation, all for the high, high price forced because you require a one-off.

      By the way, what's behind door number 1 comes with a volume discount so you can use it in several places in you operation. What's behind door number 2 comes with a volume discount of one because its a one-off.

      Choose wisely.

      • Re: (Score:2)

        by davidwr ( 791652 )

        You forgot doors 3 and above:

        Door 3:

        Actual 1970s-era chips or slight revisions of them that are still being produced: Why do I need a system with an 80486 or modern-ARM-chip with a wired or wireless Ethernet Ethernet interface when a 4- or 8-bit microcontroller and a simple wired or wireless serial interface will do?

        Door 4, more expensive, feasible only if you are ordering tens or hundreds of thousands:

        Same microcontroller and same serial interface controller but with features you don't need removed during

        • an 8-bit micro-controller won't cut it anymore.

          Everything is so precision motion-based and customizable.
          Everything needs to have comprehensive diagnostics to say what's wrong (because troubleshooting has been replaced with magical thinking).
          Everything is running at multiples of 125 microseconds, now even update rates of 10 milliseconds is too long.

          Serial (RS232) is an absolute nightmare to diagnose. RS485 is a nightmare to get them to wire it correctly. YES, RS485 as in 2 wires and a shield is still an abso

          • Spot-on. Not sure if you are being sarcastic on a couple of the points, but when you have access to second and third or fourth derivative data rather than a simple counter you have much more valuable diagnostic data to work with to streamline a process. It might not be needed continuously, but to have it at all means you get it continuously.

            Yes, it can make people lazy-- we will just tune it in the field-- but for some things a 1% delta is really important.

          • Re: (Score:2)

            by sjames ( 1099 )

            So go with a 32 bit microcontroller with an ethernet port.

        • This has always been an option, and the industry chose Door #1 years ago because it is by far the most productive and economical.

          It's not like the machinery is getting any simpler either. How much data do they send and how often do they need to report? Do they integrate with inventory systems to track usage of raw materials and other consumables? How good are the predictive wear/failure alerts?

          More precision and more complex automation are going to push those 1970s-era control systems from outdated to unwor

          • This has always been an option, and the industry chose Door #1 years ago because it is by far the most productive and economical.

            Three competing companies are on the same street; one chose door #1, the others chose door #2. One of the ones who chose door #2 has higher productivity and TCO than #1, because the engineering consultant they hired delivered what they promised. The other third company is about to be shut down and liquidated because their consultant never delivered, and after throwing good money after bad, they eventually opened door #1 but because of their debts they didn't have the cash flow to buy large enough quantities

        • Door 5 is cheaper, is clearly used for a similar purpose by companies larger than yours, but the manual in only in Chinese and the marketing documents in "English" don't make any sense.

        • Those '70s era microcontrollers easily fit on a cheap FPGA, allowing full customization.

      • As a software consultant I can inform you, you understood door 1 just fine. But you didn't really understand door 2 very well. It does actually scale, you just fell off your knowledge cliff. When you used the word "specified," that includes scaling.

      • Re: (Score:2)

        by sjames ( 1099 )

        There's a whole high volume world of microcontrollers out there that are more powerful than the chips of the '70s and available dirt cheap. There'd be even more in use but programmers who can write software for them are slightly more expensive.

        It's not just the CPU. The microcontrollers tend to have very simple requirements for support on the board. Some can even be trivially plugged into a breadboard and brought up with 5 or so wires.

    • I agree, the problem is since most developers use Windows they will design their applications on it.

      If we can get more developers in the industrial sectors to swap to something more cross-platform then they could use locked down ARM devices like Raspberry Pi's that can be swapped out more easily to check for security audits.

    • Re: (Score:1)

      by Anonymous Coward

      The systems affected are not the actual PLCs that directly control the equipment.

      These are the HMIs and programming software that runs on a windows workstation.
      (The Panel-Mounted HMIs also run typically WinCE, although the newer PV5500s run linux)
      And those are *anything but simple*.

      I run all my industrial programming software in VMs for just that reason--there are bizarre dependencies and incompatibilities between different manufacturers of this software, and even between different vintages of software from

    • Re: (Score:2)

      by tlhIngan ( 30335 )

      The problem is actually not the SCADA system itself. It's the computer monitoring the SCADA equipment!

      The meltdown and spectre patches that Microsoft released are apparently causing problems with the monitoring and configuration software that runs on the computers attached to the SCADA network. The equipment is fine.

      • A little off on your description. SCADA is Supervisory Controls and Data Acquisition. There are several parts and pieces to that and in most systems, that included the operator interface which is usually run on Windows machines.

        Yes, the equipment that interfaces with the field equipment is fine, but the operators can't see what that equipment is doing.

        It would be like saying, your car is fine and the engine is running, but your brakes, gas pedal, steering wheel all stopped responding and your windshield i

    • Re:Industrial systems should be super-simple (Score:5, Informative)

      by Rogue974 ( 657982 ) on Tuesday January 16, 2018 @02:38PM (#55940215)

      I am a controls engineer and use the software mentioned in this post.

      First, controls guys who know anything and don't get IT telling them, you must do this now, will never install a patch until vetted by the manufacturer. I actually got a notice from the vendor saying, don't install this patch 2 days after the patch was available.

      As to being more complex then they should be or simple...

      The actual controllers that run the process are extremely simple, extremely hardened and designed to run 24/7/365. PLC processors cost $4000-$15,000 depending on type and memory and they get into the hundred of meg of memory.

      Where it gets difficult is when you start using PCs to run your operator interface. There are tons of graphics, reports, trends, etc and you use software that is designed to run on Windows, which most of your operator interfaces are designed to do.

      When a patch like this hits, the operator interface or historian has issues, but the PLC running the process keeps doing it's job, you just can't see into the PLC.

      So yes and no. There are things that are more complex and that could be simplified/run separate from windows, but those start getting prohibitively expensive and the tiny bit of extra reliability is not needed. Those kinds of systems cost 2-5 times as much and the development of those systems is more expensive because there are even fewer people with experience with it. If I had experience with those systems, I would be making 70% more then I am now and I am making enough that I don't need to complain.

      • For many systems you can make away with dedicated PLC and use something like TwinCAT - turn your windows PC into PLC and have everything in one box with as much oomph and memory as a PLC could ever want. Of course for the particular system it must be acceptable that when windows goes belly up so does the PLC, doesn't happen all that often but it will happen eventually.

        • If you have a system that simple and unimportant that you can throw a desktop computer at controlling it then there will be a properly industrial hardened PLC in your price range. No seriously some proper PLCs from the vendor mentioned in the post run at a lower cost than any computer you can throw at it. Just configure and provide it with 24V.

          But chances are if you're having problems with the performance of a Wonderware Historian you're problem is that you have 5000 analogue instruments on site and you're

      • Re: (Score:2)

        by sphealey ( 2855 )

        The problem, as Target Corporation learned the hard way, is that control systems and their supervisory PCs are often the attack gateway into corporate financial systems. Yes, I'm sure your entity's control systems are never connected to the Internet and are invulnerable - just as Target's HVAC controllers were standalone, not connected to the Internet, and would never be allowed to communicate with the credit card processing system. Problem is they weren't, they were, and they did. If the control system

        • Agreed. Even air gapped, a system still needs to have security patches applied.

          It doesn't matter how great your protocols are to limit access, included in your policy has to be patching. There are many things you have to consider when securing a network with life critical things attached to it. When and how you apply those patches is just as important as any other parts of the policy to secure your network.

          Are my systems at risk from Meltdown and Spectre because I have not patched? Yes. Due to other la

    • Probably at least somewhat true that you don't need much of a computer for many industrial control tasks. And even when not true, do the systems have significant sensitive information that needs to be hidden from other users? Is there any point in patching the software against spectre/meltdown if the patches cause trouble and the malware doesn't represent much of a threat to those systems?

      OTOH, getting infrastructure OFF the damn internet as quickly as it can be done seems like a very good idea even if do

      • Yes, like you were agreeing to, to run an industrial controls systems you don't need much power, it is just the SCADA (operator interfaces and historian) where you start needing a lot more and it is very difficult to do without getting into the PC and server areas.

        As to controls systems being on the internet, yeah, those people are idiots or dealing with stuff that is non proprietary, non life threatening.

        At my place, there is not outside logging into the system. They need troubleshooting, I drive in. Oth

    • Or they could be isolated and unpatched. An airgap is the best security, always.

      • This is an interesting question. Air gapped is the best solution, but not always acceptable.

        Some places need to allow remote access to their controls systems for troubleshooting purposes because they have few experts and it is impractical to fly your controls engineers all over the place.

        Even more common is the need to get historical and inventory information up to the business network real time so people can make proper decisions. Security best practices talk little about air gapping because almost every

        • Re: (Score:1)

          by davidwr ( 791652 )

          Even more common is the need to get historical and inventory information up to the business network real time so people can make proper decisions.

          This is where a one-way air gap can come in handy:

          Have the "secure" side continuously report real-time data 24/7 to a "less secure" device for recording/reporting over a one-way channel. Use VPNs or other controls to give access to the "recording/reporting box" as needed.

          If the reporting/reporting box gets compromised, at least it can't directly leapfrog back to the "secure side."

  • a lot of the manufacturing stuff is stuck in the direct hardware access ideas of dos / win 3.1.
    Someone needs to do an DirectX / opengl like layer for this stuff to use.

    • a lot of the manufacturing stuff is stuck in the direct hardware access ideas of dos / win 3.1.

      That's not necessarily a bad thing. Manufacturing equipment generally doesn't benefit greatly from having the latest and fanciest do-dads and gee-gaws. It needs to be reliable above all else in most cases. Well made manufacturing equipment mostly should never need to be patched. There are some exceptions but they are rare. You do run into some networked CNC and robotics that needs to be more up to date but the presses that my company runs really just need to go up and down. More complexity or features

    • Re: (Score:2)

      by RobinH ( 124750 )
      That's what OPC UA is, from my understanding. I never like original OPC due to the performance hit, but I understand that OPC UA is much better.

  • VMware pulled some of their patches (Score:3)

    by El Cubano ( 631386 ) on Tuesday January 16, 2018 @01:38PM (#55939705)

    VMware pulled some of their patches

    Note: ESXi patches associated with VMSA-2018-0004 have been pulled down from the online and offline portal.

    ...

    For ESXi hosts that have not yet applied one of the following patches ESXi650-201801402-BG, ESXi600-201801402-BG, or ESXi550-201801401-BG, VMware recommends not doing so at this time. It is recommended to apply the patches listed in VMSA-2018-0002 instead.

    ...

    For servers using the Intel Haswell and Broadwell processors (see Table 1 for the specific list of affected VMware vSphere supported Intel Haswell and Broadwell processors) that have applied ESXi650-201801402-BG, ESXi600-201801402-BG, or ESXi550-201801401-BG VMware recommends the following:

    ...

    VMware is working closely with Intel and the industry to come to a quick resolution of this Intel microcode issue and provide an update to our customers as soon as possible.

    Oops!

    reference [vmware.com]

    • Re:VMware pulled some of their patches (Score:4, Insightful)

      by El Cubano ( 631386 ) on Tuesday January 16, 2018 @01:46PM (#55939783)

      I guess I should have finished my thought.

      It's not just industrial control systems, but hypervisors, and plain old systems too. It sees like this is an object lesson in how speed (in terms of releasing a fix) comes at a cost of performance/quality. I know people were all in a panic once Meltdown and Spectre became public, but this wasn't just fixing a SQL injection vulnerability in Rails or Django. This fundamentally affected the execution of nearly every instruction to go through affected CPUs.

      I suspect that the severity and publicity made a more organized roll out with extensive beta testing impossible for just about every vendor that had affected products.

  • Stuxnet is now crashing (Score:3, Insightful)

    by Anonymous Coward on Tuesday January 16, 2018 @01:38PM (#55939709)

    Now things like Stuxnet won't be able to infiltrate as easily. WTF are these things doing connected anyway, and if not connected why do they need the patches? And don't get me started on Windows...

    • WTF are these things doing connected anyway

      So to answer your question from multiple points:

      Stuxnet : Stuxnet did not infect a connected system. It was spread by direct attack via USB.
      Connected : A wonderware historian serves the sole purpose of storing long term trends of your entire control system. This is very useful data ... if you have access to it. This is very sensitive data which some governments will require you to store in realtime ... offsite.

      Now to make a point: On a well setup and secure site the Historian will sit on a different network

  • So the meltdown patches are themselves causing meltdowns? Isn't it ironic! (Doncha think?)

  • People use windows for industrial automation??? Scary world!

    • Yes, absolutely, most industrial automation are run on Windows.

      No, it isn't that scary. It can be, but if properly implemented with the right security in mind, you can keep the system up and running reliably.

      As stated below, the windows machines are used for the operator interfaces and to record information. The things that actually controls the process are different and unaffected by this and the screwy things with windows.

      I am a controls engineer, i.e. program, spec, maintain, industrial controls system

    • Linux has pretty much zero footprint in industrial automation, if you have a full OS running somewhere its going to be windows.

  • Toldja so... (Score:5, Insightful)

    by GerryGilmore ( 663905 ) on Tuesday January 16, 2018 @01:54PM (#55939849)
    From the very beginning, I've tried to get everyone to pause the Panic Parade, but nnnnnooooooo. To try to address probably the most complex vulnerability yet discovered (it took over 20 YEARS for this to be found) that also requires you to already be running malware on your system, people are flashing new BIOSes, patching kernels and generally behaving like idiots. Slow FT down, folks! Let the CPU and OS experts have a real shot at minimizing the risk, without killing our production systems, FFS!!

    • Yeah, pretty much everything GerryGilmore said. This knee-jerk reaction to a pretty obscure flaw is way overboard.

      I personally don't want my CPU's branch prediction gimped because some other idiot can't keep his web browser away from malicious sites.

      The only panic that should be realistic and warranted is big cloud VM providers concerned these attacks could compromise tenants on shared systems. Patch that shit into oblivion as far as I'm concerned, but get your grubby patches off my desktop. I don't want

  • Also Beckhoff TwinCAT 3 (Score:5, Interesting)

    by RobinH ( 124750 ) on Tuesday January 16, 2018 @02:08PM (#55939973) Homepage
    We received a notification from Beckhoff to avoid these patches for TwinCAT 3 until they would patch their runtime to be compatible. We update through WSUS so we were able to do that. Beckhoff themselves urge you *not* to install Windows Updates on their control system PCs even though they bill their product as part of the "Internet of Things" and play up the connectivity of everything. They're hypocrites, but Rockwell did the same thing when we used their product. They wouldn't warranty their software if you installed anti-virus on the same server as their historian product.
    • It is not hypocritical for them to warn against the dangers of applying Windows updates, especially for industrial applications. Microsoft has a well established pattern of changing functionality without warning via patches and they Bork systems on a regular basis.

      • Re: (Score:2)

        by RobinH ( 124750 )
        It's absolutely hypocritical of them to sell a Windows PC-based industrial control system, and instead of maintaining their product and testing it with each new Windows Update, they just put out a blanket statement to turn off Windows Updates and put it behind a firewall, and then sell their product based on the connectivity it provides. I'm OK with delayed patch installation and extra security measures, but every patch needs to be tested by them and certified for installation. They have no mechanism for
        • Most industrial systems are built once and then left well alone to not muck anything up, periodic Windows updates and accompanying mandatory restarts are pretty much a non starter in most cases. If it works, don't touch it, is the wisdom when it comes to industrial systems.

          • Re: (Score:2)

            by RobinH ( 124750 )
            That is the old model before all the data collection requirements arrived. The new model needs data connectivity, so patches are a must, or else you need expensive, complicated and difficult-to-implement things like data diodes.

        • Quote, "I'm OK with delayed patch installation and extra security measures, but every patch needs to be tested by them and certified for installation. They have no mechanism for doing that at all."

          They actually do have a mechanism for testing and releasing what patches are acceptable on their systems. This articles talks about Wonderware and Rockwell systems, both of which I use on a daily basis as an end user. Both make available a list of what patches they have tested and vetted out for their systems.

  • Industrial systems don't have as much spare room (Score:5, Insightful)

    by plague911 ( 1292006 ) on Tuesday January 16, 2018 @02:12PM (#55940001)
    I have never worked on industrial systems, I did work on some large scale defense equipment. One of the design considerations is cost, in order to minimize cost, you match the components spec to the semi-well defined performance need. No need on buying a V12 when a V6 will do...... Now I am not saying you don't build in some buffer, but the MASSIVE performance hit required by these patches could easily blow the given performance buffer out of the water. I could easily see how billions of dollars worth of industrial systems simply will not be able to patched due performance cost of the patches. Additionally given the age/design of the systems there is no way to conveniently upgrade the systems.
    • Your whole diatribe is ridiculous. The parts of the system to which you refer aren't even the ones affected, nor would they be the way you describe it. These are the GUI based interfaces that are experiencing the issue, not the core system itself as you falsely assumed.

    • I could easily see how billions of dollars worth of industrial systems simply will not be able to patched due performance cost of the patches.

      Doubt it. The systems themselves are unaffected by this mess and don't need patching in any case (controlled access and all). What is being patched here is a handful of database servers. Spend the $3000 to buy a better one. In the grand scheme of "industrial control" not only will it not break the bank but you can most likely pay for it using petty cash.

  • If your control systems weren't connected to the Interwebz so you could save a little money on payroll, you wouldn't have to apply the patch.

    • Yeah, not so much.

      Controls systems not connected to the internet still need to be patched and maintained because there are vectors of attack that can still get across an air gap.

      Yes, patching isn't as important, but you still have to patch for security and just to be able to stay compliant with software revisions of the software you are using.

      FYI, I am a controls engineer, that means I do this for a living. I use the software mentioned that this crashes, but it didn't hit me because I would never apply a p

    • Says the imbecile who is not aware of how security works. Just ask the Iranians about how not connecting ICS systems to the internet worked. Yes there are plenty of dumb managers out there who would want to connect these systems to the corporate network (and thus the internet) to save a buck but then there are methods of attack that can work around even an air gapped one. The people who are actively targeting these systems are not just your run of the mill script kiddies, or computer crime orgs, but often a

  • User:*Computer install patch and do regression test.

    Computer: The company has now regressed

  • The hazards of monoculture (Score:4, Insightful)

    by plopez ( 54068 ) on Tuesday January 16, 2018 @02:33PM (#55940157) Journal

    relying on one piece of tech is as bad as relying on one food crop.

  • If you're running an important control system using Windows, you're doing it wrong.

    • Says the guy who has never created industrial anything. Not that it wouldn't be nice to use Linux, but reality is that Linux might as well not exist on industrial scene, all the software for pretty much anything is Windows only, so good luck working with your Linux based control system.
  • not to patch shit just because some code money says to. Security and functionality are in direct opposition to each other. Increasing security lowers availability and increases the number of points of failure. This is a trade that real engineers (as opposed to software weenies) make in the presence of limited computational resources. Things that face the public and run unattended in remote installations? Err on the side of security. Things that never see the light of day and run inside a steel vault with tw
  • I've just installed it in the nuclear reactor control system. The IT security department inist.......auuugh!

Slashdot Top Deals

What is research but a blind date with knowledge? -- Will Harvey

Close