World's Largest Chip Maker Will Lose $250M For Not Patching Windows 7 Computers (networkworld.com) 108
A major virus infection forced the closure of Taiwan Semiconductor Manufacturing Company (TSMC) factories last weekend..." writes Slashdot reader Mark Wilson, noting that it's the largest semiconductor manufacturer in the world, selling chips to Apple, Nvidia, AMD, Qualcomm, and Broadcom, and "responsible for producing iPhone processors."
Now Network World reports: The infection struck on Friday, August 3, and affected a number of unpatched Windows 7 computer systems and fab tools over two days. TSMC said it was all back to normal by Monday, August 6. TSMC did not say it was WannaCry, aka WannaCrypt, in its updates, but reportedly blamed WannaCry in follow-up conference calls with the press.... The company said this incident would cause shipment delays and additional costs estimated at 3 percent of third quarter revenue. The company had previously forecast revenues of $8.45 billion to $8.55 billion for its September quarter. A 3 percent loss would mean $250 million, though actual losses may come out lower than that. Still, that's a painful hit. TSMC also said no customer data was compromised....
TSMC isn't directly to blame here; someone [an infected production tool provided by an unidentified vendor] brought WannaCry into their offices and behind their firewall, but TSMC is still culpable because it left systems unpatched more than a year after WannaCry hit.
Now Network World reports: The infection struck on Friday, August 3, and affected a number of unpatched Windows 7 computer systems and fab tools over two days. TSMC said it was all back to normal by Monday, August 6. TSMC did not say it was WannaCry, aka WannaCrypt, in its updates, but reportedly blamed WannaCry in follow-up conference calls with the press.... The company said this incident would cause shipment delays and additional costs estimated at 3 percent of third quarter revenue. The company had previously forecast revenues of $8.45 billion to $8.55 billion for its September quarter. A 3 percent loss would mean $250 million, though actual losses may come out lower than that. Still, that's a painful hit. TSMC also said no customer data was compromised....
TSMC isn't directly to blame here; someone [an infected production tool provided by an unidentified vendor] brought WannaCry into their offices and behind their firewall, but TSMC is still culpable because it left systems unpatched more than a year after WannaCry hit.
Yep, that's what you get (Score:3, Insightful)
for not patching your systems.
Re: (Score:3)
The problem here us unlikely to be that IT was too lazy to upgrade or unwilling to patch. Quite the opposite is generally the case.
Vendors that supply process control systems will certify exactly what can and cannot be loaded on these systems including patches. It can take years to get a new patch certified from the vendor. And if you load anything uncertified you are taking on that entire liability hit and lose support and such. That's a career limiting move.
Oh and Windows 7? Not too bad, There are
Re: (Score:2)
So what's to do? Would it be possible to have each legacy system run inside a sandbox, VM or VM-lite kind of thing, maybe like Sandboxie for Windows but industrial strength, and you make a copy of the sandboxed image every day. If a virus infects the guest OS, you simply go back a few snapshots. If the virus hasn't wiped or encrypted the application-generated data files, you can restore those from the latest sandbox or snapshot.
Is there anything obviously missing in this scheme?
Re: (Score:2)
The most likely result of that line of inquiry is going to be "Must be run on vendor supplied hardware" and "Vendor does not certify to run in a VM."
Also just to make it more fun, taking it offline to do a backup shuts down a production line and must be scheduled once a quarter or once a year.
Hmmmmm, "what's to do"
Probably nothing until manufacturing via 3d printing and general purpose robotics becomes competitive with classical manufacturing. Not because they are better or worse, but more because once yo
Re: (Score:2)
I imagine with $250M to lose over two days TSMC could easily say Hey Vendor certify your stuff to run in a VM pronto. Vendor would do it, unlike porting their app to Linux. Would they not, realistically?
As for VM, can you make a correct VM image backup while the VM is running? Seems to me that could be done in the background without affecting production.
Re: (Score:2)
Given the cost to change vendors, probably not.
Re: (Score:2)
If it's process control, a VM probably won't do. The software will likely be talking directly to some bit of hardware and any stuttering on the part of a VM passing things through to real hardware would be a problem.
Best you can do is keep the prosess control machines on an air-gapped LAN and hope it doesn't get cooties if you have to temporarily connect to the outside or connect a laptop fpr updates.
Re: (Score:2)
VFIO
Re: (Score:2)
VFIO is helpful for performance nd security, but still might cause problems if there are hard deadlines.
Re: (Score:2)
"for not patching your systems."
Perhaps their machines didn't have the chips to upgrade to the latest, greatest Windows version.
You know, the cobbler's kids are barefoot.
Re: (Score:2)
The problem in reality was not that they did not patch their airgapped system, is they breached airgap by allowing hardware in with software installed, bad mistake. You airgap a system, than thieving is airgapped, including new hardware and they way new hardware is airgapped, is it is supplied free of software. The software comes in separately and is scanned and checked and then installed on the new hardware inside of the airgap, common fucking sense, or at least it should have been.
Airgap requires that ne
Re: (Score:2)
The question here is "Who is 'they'?". It's quite likely the configuration is specified by the vendor of the $$EXPENSIVE$$ niche machinery. And they aren't going to change their specs, because, since that machinery is expensive, they don't have any old models to test on. And possibly not anyone currently expert in that particular model. (They're concentrating on the next generation model. "Want to order one? You have have it for beta testing on your production line in only a couple of months.")
Re: (Score:2)
I thought chip fabs worked hand in hand with the producers of the equipment to produce a solution that would work for them. At some point during specification, quoting and the like, the question of OS should have come up, and they should have specified something better than Windows.
Re: (Score:2)
They probably do work hand in hand with them, and then support it for the life of the equipment. The problem is when the lead engineer who wrote 90% of the software, in a poorly documented and even more poorly maintainable design, left the company, or got hit by a bus or reason X. So sure, do you want to change feature X like increase the max RPM of the motor by 15RPM to get rid of some harmonic vibration? Yeah we can do that. The problem comes when he leaves and you can't do something important without his
Re: (Score:2)
Oh come on. This is slashdot where I see IT professionals proudly say they don't patch with a smile.
I want to say told ya so.
Re: (Score:2)
Well, that's why I've heard Intel still used VAX/VMS to run their factories until at least recently.
Wrong headline (Score:1)
``World's Largest Chip Maker Will Lose $250M For Using Known-Vulnerable Operating Software''
The correct conclusion is that windows just isn't suitable to run multi-billion operations with. As long as you ignore that reality, you leave the door open to other parties to take advantage of that.
Re: (Score:2)
Re: (Score:3)
Depends on the price to switch to a system that isn't so insecure.
Where I work, we tried switching about fifty servers to Linux, but it failed due to the fact we couldn't find people that knew what they were doing for minimum wage. The two high school drop-outs that work for minimum wage do an OK job with keeping those Windows servers running. Windows is acceptable since our customer SLA is 95% so I think we can have almost five hours of downtime a week. Of course we often exceed that amount of downtime
Re: (Score:1)
Expert for what? For the most part a linux guy doesn't have to do nearly as much as a windows guy. Windows fellas need to run around like they just crapped themselves 24/7 to keep that big jenga tower of interdependant hack code which is ms windows together.
Ah you guys know your stuff though. It will all be okay, its not like these systems run operations involving salaries and materials that run into the millions of dollars of cost, nooooooooo, they are just toys that the folks with glasses use, we'll gi
Re: Wrong headline (Score:2)
Why didn't you assign the project to one of the high school students? If that's their level of competence, you'd be better off with the sane secure defaults on a Linux. It's a learning project for them, and dirt cheap R&D for you.
Re: (Score:1)
Depends on the price to switch to a system that isn't so insecure.
Where I work, we tried switching about fifty servers to Linux, but it failed due to the fact we couldn't find people that knew what they were doing for minimum wage. The two high school drop-outs that work for minimum wage do an OK job with keeping those Windows servers running. Windows is acceptable since our customer SLA is 95% so I think we can have almost five hours of downtime a week. Of course we often exceed that amount of downtime because of Microsoft-created problems, but the lost customers cost less than a Linux expert would cost.
These guys apparently found people that knew what they were doing for minimum wage and the result is...
Re: (Score:2)
Where I work, we tried switching about fifty servers to Linux, but it failed due to the fact we couldn't find people that knew what they were doing for minimum wage.
You should be paying more than the minimum wage, and if you don't, you deserve what you get for your money. Which is Windows. You should go out of business and let someone competent take your place.
Re: (Score:2)
Thing is, on linux it does tend to mess up your hardware to the point of bricking it, whilst on windows, well, from personal experience, it didn't happen that much.
try it lmsensors on linux
or on windows, some program to read out temp and set fan values.
I've been running gkrellm on all manner of different hardware for a dozen years, and I've never had it—or any other Linux software—brick a machine. I interpret your assertions to mean that you know even less about what you're doing than I do, and I am by no means what I'd consider an expert.
Re: (Score:2)
Whats wrong with lmsensors?
it8686-isa-0a40
Adapter: ISA adapter
CPU Vcore: +1.31 V (min = +0.00 V, max = +3.06 V)
+3.3V: +3.33 V (min = +0.00 V, max = +5.05 V)
+12V: +12.17 V (min = +0.00 V, max = +18.36 V)
+5V: +5.01 V (min = +0.00 V, max = +7.65 V)
Vcore SOC: +1.00 V (min = +0.00 V, max = +3.06 V)
CPU Vddp: +0.92 V (min = +0.00 V, max = +3.06 V)
DRAM A/B: +1.38 V (min = +0.00 V, max = +3.06 V)
CPU fan: 5625 RPM (min = 0 RPM)
SYS1 fan: 0 RPM (min = 0 RPM)
SYS2 fan: 0 RPM (min =
Re: (Score:2)
This is a custom machine configuration. There are lots of custom configured Linux machines that can't be updated. Your desktop is not a valid comparison.
OTOH, if they can't do something like run it in a VM, then the problem isn't the OS, it's the licensing agreement. Or possibly the design. That said, time sensitive things often can't be run well under virtualization. And are often sensitive to even minor system upgrades. So it could well be a combination of time sensitivity and a CYA licensing agreem
Re: (Score:2)
Re: (Score:2)
Last I checked Linux has vulnerabilities too that any competent administrator would patch. FYI I have seen SuSE services use for hosting phishing sites with the customer not having any idea due to a rootkit.
Rootkits were invented on Unix. Where do you think the term ROOT came from?
Re: (Score:3)
I know of a Fortune 500 company that will move to web-terminals after Win7, exactly because of all these problems. They found that qualifying Win7 and dealing with problems from all the updated and lack of security was more expensive than just making all their stuff (mostly custom applications) web-only in their intranet. There will not be any Win10 except by special permission.
Re: (Score:2)
What is the size (employee count) of the company by chance?
A word about these computers... (Score:5, Informative)
I once worked on a medical device where each and very build installed MUST be a bit-perfect replication of the original. Any new release went through horrific levels of qualification and then IT had to be bit-perfect until the next release.
The typical "patch Tuesday" crap just cannot work in these environments.
Re: (Score:2)
Or in other words, MS Windows is just about the worst OS choice possible for such applications.
Re: (Score:2)
When I was working at a SCO UNIX shop,
Re: (Score:2)
Ah, yes. I have run into that stupidity as well. Many people just do not understand that maintenance is the majority of the cost in OS usage. Fortunately, our customers are usually migration from some commercial UNIX to Linux, and that is pretty painless. Also RHEL is maintaining old software with security and crash fixes forever, so updates are low-risk.
Re: (Score:2)
If an attack is targeted the choice of OS is quite irrelevant. This attack however didn't look targeted, but then also ... wannacry. I would wager that the evening janitor they entrusted to set this up in his spare time would have done an even poorer job with a more esoteric OS.
Re: (Score:2)
Sure. Or rather, as long as the attacker has the skills, it is. But would anybody in their right mind do a targeted attack against a company, that could put a $10M price (or higher) on their head without any problems? It is good criminal practice to stay an annoyance and to not become a real threat. Competent criminals understand that.
Re: (Score:2)
But would anybody in their right mind do a targeted attack against a company, that could put a $10M price (or higher) on their head without any problems?
Yes, because this is the real world and not some funny action movie staring Steven Segal.
Corporate espionage and corporate sabotage are a very real thing that happens constantly and sometimes is even state sponsored.
Re: (Score:2)
You seem to be the one in the movie...
Re: (Score:2)
You could be right. After all someone is telling me that something that happens constantly doesn't actually happen. Either I'm in a really poorly written movie, or you're gunning for a republican presidential nomination.
I declare all of history fake news from this point on wards.
Re: (Score:2)
I would tell you your data is flawed, but you are thoroughly caught in your filter-bubble, so that is just a waste of time. You are _incapable_ of seeing what is.
Re: (Score:2)
I would tell you your data is flawed, but you are thoroughly caught in your filter-bubble, so that is just a waste of time. You are _incapable_ of seeing what is.
Yep like I said, all of history if fake news to you nutters.
Re: (Score:3)
Basically, you don't know, and that's the rub. Let's take as an example the latest set Spectre/Meltdown patches. These are known to affect I/O performance (heavily-syscall-dependent) to a degree anywhere from 5-30%. Given that this is ONE patch, the same basic rules apply in, essentially, what are semi-real-time systems. That is, for each and every patch, you must apply the entire set of QA tests, which takes a lot of time and money. Perfo
Re: (Score:2)
About 20 years you could buy these little PCI cards that had some kind of BIOS ROM that prevented permanent changes being made to the hard drive. Writes were redirected to free space, and when the machine rebooted they were discarded.
They were popular with internet cafes. Hit the reset button and the machine went back to the default state, no matter how many viruses the previous user managed to get infected with.
Re: (Score:1)
Re: A word about these computers... (Score:1)
A classic example, that actually has hit many companies, is DCOM: A Microsoft technology that has something to do with setting up RPC connections (usually from programs written in VB or dot net). It is provided by the OS itself, and uses the OS security setup for authentication.. There have been several patches to fix security holes in it, which at the same time caused client apps to start working differently or not at all. Most shops ended up abandoning the technology and switching to various XML based rem
Re: (Score:2)
Because of their VERY finicky nature (and usually being designed to be used on a closed intranet), they almost NEVER apply post-production patches.
Medical device and process control are two very different systems. Process control systems most definitely do get patched. Not instantly, they go through vendor approval first, but they most definitely do get patched.
Re: (Score:2)
Yes, but...
The questions are "How many of the model were sold?" and "How long since it's been under active development?" and "What's involved (cost) in keeping an idle system around?" and "How many experts in this particular model does the manufacturer currently employ?".
I suspect that combining the answers to those questions would yield "The manufacturer will not support ANY changes in the supplied configuration.".
Re: (Score:2)
The screw-up here is using an OS that cannot be professionally operated...
Re: (Score:2)
... like MS-DOS.
Re: (Score:2)
Right because Linux is so perfect and secure and never has been hacked before or needs patching.
Re: (Score:2)
You are saying "Linux", I did not. One advantage of Linux is that it usually does not break on update, though. That is, before systemd. But there are other alternatives.
Re: (Score:2)
?? Have you seriously ever ran a distro without updating? No distro in existence can still function after 2 updates. It always requires a re-install because it lacks an ABI driver model which every other OS has for decades now.
Re: (Score:2)
I have automatic updates every 3 days enabled on some of my servers. No problems in about 12 years now. Were does this stupidity about "no distro can function after 2 updates" come from? Are you utterly clueless what you are talking about?
Save a penny, lose a million (Score:2)
The classical effect of mindless bean-counters that do not understand risk-management at all. Pathetic. And, since further up you usually find the same bean-counters, those that messed up massively here will likely not even be fired.
Re: (Score:2)
Re: (Score:2)
You assume I criticize them not patching. You are wrong.
No, YOU don't understand fab economics (Score:2)
You never want to take a wafer fabrication plant offline for unscheduled maintenance, because having a line down costs you $1 to $10 million an hour while you're down. Worse, if you take it down for anything but regularly scheduled maintenance, you have to re-qualify the tool, which can take weeks.
And if you have to take all your etch tools, or all your metal deposition tools, or all your steppers down, because they all run on the same version of Windows 7, then you're burning through tens of millions of do
Re: (Score:2)
You assume I criticize them not patching. That is not correct.
Re: (Score:2)
This is probably some Windows front-end monitoring and configuration software for some SCADA or SCADA like systems. Since Windows has a tendency to break on updates, they probably just isolated the network and there was some report that an outside supplier brought the malware in by being sloppy.
The sane thing would of course be to put some hardened OS with low-risk patching on these machines (e.g. a hardened RHEL) and still have them on an isolated network. Would now likely also have been the cheaper thing.
Not lost (Score:4, Informative)
Just delayed until the next quarter.
Also, lower revenues are not money "lost".
Also, a newer story says it's $170 M, (2% of revenue), not $250M: https://digitimes.com/news/a20... [digitimes.com]
But it wouldn't be a modern news story without a bunch of exaggeration and misunderstood info, would it? The important thing isn't the correct facts, the important thing is to point and laugh at someone's misfortune. Because news...
Microsoft will use this (Score:2)
Re: (Score:2)
The security risks of using Windows 7 outweigh the time wasted de-bloating Windows 10.
What about the privacy risk of Windows 10, and the fact that it is still riddled with vulnerabilities? [cvedetails.com] Just stop abusing yourself and install Linux. If you absolutely must run Windows then run it under KVM. I hear tell that Windows on KVM is actually more efficient than Windows running on the metal, perhaps because of more efficient file system and block device handling.
Oh man, poor TSMC! (Score:2)
If I had lost $250 million, I would WannaCry too!
Re: (Score:2)
That's a good point. Especially the " they failed to implement protocols for bringing other systems into the environment.", as there are many reasons why they may not have been able to patch the system.
The problem is not banning Windows (Score:2)
Google learned this lesson and banned Windows from inside their network, a Windows machine can now be connected to the network only with VP approval. Other organizations are perhaps more stupid.
Windows is also banned from the world's financial systems after the LSE fiasco. But US Navy is too stupid to ban Windows even after towing that missile cruiser [wikipedia.org] back to port. It should be illegal to use Windows in medical devices, until it does become illegal it should should be a lucrative income source for ambulance
Re: (Score:1)
even after towing that missile cruiser [wikipedia.org] back to port.
That case is ancient. It's Windows 4.0 old. It's Rex Ballard advocacy old. It's tired and anybody with a clue remembers people citing it ten years ago when it was already extremely outdated and old.
Re: (Score:2)
even after towing that missile cruiser [wikipedia.org] back to port.
That case is ancient.
Of course it is, but nothing changed after that, that has to tell you something.
The LSE fiasco is not ancient, Windows is still banned from the world financial system. Not to mention the top 500 list. Islands of sanity. We need more.
Re: (Score:1)
Nothing has changed since Windows NT 4.0?
Maybe in your world.
Re: (Score:2)
Nothing of substance has changed in the Microsoft world. Especially, attitude has not changed, you are living proof of it. And for your information, not a lot has changed in the Windows kernel since Windows NT either, but I would not expect a random Microsoft troll such as yourself to know a whole lot about that. Linux on the other hand changed radically (while preserving external interface stability) in that same period.
One thing in particular has not changed about Microsoft and its products: they remain a
Re: (Score:1)
See, Microsoft has not changed a bit and Microsofties are still the same, except maybe more bitter now. Thanks for the demonstration.
Microsoft causes Chip Maker to lose $250M (Score:1)
Was $250 million more than the cost of updating? (Score:2)
Given the size and numbers, is $250 million more or less than the cost of keeping their infrastructure up to date?
And even after this costly mistake by a vendor, just keeping their systems tightly locked down and having much better controls over who or what gets plugged into their network may be far cheaper than updating everything.
Given that they were back up and running quickly, it does appear that they have everything locked down and backed up. I expect they knew what the risks where and are and will upd