Forgot your password?
typodupeerror
Bug Technology

Botched Security Update Cripples Thousands of Computers 274

Posted by samzenpus
from the houston-we-have-a-problem dept.
girlmad writes "Thousands of PCs have been crippled by a faulty update from security vendor Malwarebytes that marked legitimate system files as malware code. The update definition meant Malwarebytes' software treated essential Windows.dll and .exe files as malware, stopping them running and thus knocking IT systems and PCs offline, leaving lots of unhappy users and one firm with 80% of its servers offline."
This discussion has been archived. No new comments can be posted.

Botched Security Update Cripples Thousands of Computers

Comments Filter:
  • by Frosty Piss (770223) * on Thursday April 18, 2013 @12:28AM (#43479735)

    ...is all I use these days.

    Of course since Windows is "out of favor" here, one does not necessarily mention that Microsoft's "Security Essentials" is easily as good as most commercial Windows anti-malware packages, and much more "light weight". And free. And yes, everyone knows that Microsoft purchased the original technology (so what?) ...

    • by H0p313ss (811249) on Thursday April 18, 2013 @12:37AM (#43479779)

      Same here. But you should be aware that every time this topic comes up MSE is highly praised by Slashdotters.

    • by tuppe666 (904118) on Thursday April 18, 2013 @12:52AM (#43479827)

      Microsoft's popular Security Essentials anti-virus software has failed to gain the latest certificate from the AV-TEST institute. http://www.theverge.com/2013/1/17/3885962/microsoft-security-essentials-fails-anti-virus-certification-test [theverge.com] "In antimalware testing against a range of products, AV-TEST failed to certify AhnLab V3 Internet Security 8.0, Microsoft Security Essentials 4.1, and PC Tools Internet Security 2012 out of a total of 25 different vendors. Microsoft's own anti-virus software failed to adequately protect against 0-day malware attacks, scoring an average of 71 percent vs. the industry average of 92 percent."

      Nobody cares whether its original they care if it works.

      • by bloodhawk (813939)
        it really only did average on the zero day stuff, which is not the strong point of essentials. on the known malware it still does very well. the tests by AV-Test really don't provide a good way for the average user to judge products as most are not under attack from zero day malware and viri.
      • by Frosty Piss (770223) * on Thursday April 18, 2013 @01:03AM (#43479873)

        "AV-TEST institute" is well known to require financial investment for a top rating, their recommendations - such that they are - are highly suspect.

        • Re: (Score:2, Insightful)

          by minus9 (106327)

          If their results can be bought, Microsoft would have bought them.
          • by AmiMoJo (196126) *

            Why? They are not selling anything. MSE comes built in to Windows 8 and is a free download for their older systems. It exists to reduce their support costs and make Windows itself more secure, more or less transparently to the user. It doesn't try to scare you with dire warnings about tracking cookies and there is no up-selling or paid version.

            MSE isn't competing with anti-virus software so there is no reason to try to game these kinds of tests.

      • by Dahamma (304068) on Thursday April 18, 2013 @01:06AM (#43479879)

        The problem is the solutions that may do a bit better catching the 0-day malware are also the ones that are so heavyweight they noticeably affect the performance of your system. There is a tradeoff at some point between resource usage and coverage. One thing MSE definitely has going for it is it doesn't badly degrade performance like McAfee, Norton, recent AVG, etc do.

      • by tlhIngan (30335)

        OTOH it seems every one of those "passing" AV solutions at one time or other have marked a critical Windows file as a virus and made the system unbootable. Now, whether or not you can recover from that or reinstall from scratch is a good question.

        MSE fails because it's less strict, probably because you don't want it to quarantine some valuable Windows file that makes it unbootable.

        Sure Microsoft could crank up the heuristics and mark more malware, but you risk accidentally tagging a legit file - and the inc

      • Nobody cares whether its original they care if it works.

        But only if it doesn't hose your system in the process. MSE might not be the most water tight security app out there, but is hits a pretty nice sweet spot for 'good enough" security as well as "low enough" impact on performance. It's also free which makes it pretty hard to beat for a client based malware solution.

        • by Electricity Likes Me (1098643) on Thursday April 18, 2013 @02:15AM (#43480087)

          Basically "stop doing stupid things with your computer".

          Why a firm needed Malware Bytes on it's servers in the first place is the real question here.

          • Re: (Score:2, Redundant)

            by BulletMagnet (600525)

            Basically "stop doing stupid things with your computer".

            Why a firm needed Malware Bytes on it's servers in the first place is the real question here.

            I was wondering this exact same thing. IT Manager Fail.

          • by AmiMoJo (196126) *

            The services that servers provide are sometimes vulnerable to infection. Say someone found a way to create a new SQL based worm, for example. If it is a file server you might also want it to scan said files periodically. Anti-virus for servers is a good idea, although perhaps you were questioning the user of Malware Bytes in particular in which case I might agree it seems like a somewhat odd choice.

          • by couchslug (175151)

            Why a firm runs WIndows on its servers is the real question here.

      • I don't use MSE to protect my PC from 0 day exploits. I don't consider my online behavior to be that risky, and so far that assumption has held true. MSE is there mainly for the random drive-by attacks that can still happen. Better 0 day detection also results in more false positives, and this is definitely something I don't want when I'm not even engaging in risky behavior to begin with.

        Having worked as a shop tech for years my rule of thumb has been that if it's a single user PC and they are a responsible

    • by inflex (123318) on Thursday April 18, 2013 @01:08AM (#43479883) Homepage Journal

      All I use and recommend now as well. Previously good AV suites have become pointlessly (for the consumer) bloated and I'm having a higher occurence of machines being bought in with faults explicitly attributable to the AV suites.

      I'm no fan of Microsoft, but I have to say that MSE does tend to do an acceptable job given that inevitably all AV suites let stuff slip past.

      • Re: (Score:2, Insightful)

        by Anonymous Coward

        All I use and recommend now as well. Previously good AV suites have become pointlessly (for the consumer) bloated and I'm having a higher occurence of machines being bought in with faults explicitly attributable to the AV suites.

        Which is why, over a year ago, I tried out MSE, found that (at least, back then) it was as good as the usual freebie AV offerings, and installed it on a number of customer PCs and laptops.

        I'm no fan of Microsoft,

        I got a serious amount of stick for going the MSE route, I've cordially detested Microsoft and it's unholy offerings since DOS 3.2

        but I have to say that MSE does tend to do an acceptable job given that inevitably all AV suites let stuff slip past.

        And this is the thing, '..inevitably all AV suites let stuff slip past
        I've had infected machines back to me for disinfection which had been running fully up to date AV suites (both free and comm

    • by donscarletti (569232) on Thursday April 18, 2013 @02:04AM (#43480053)

      ...is all I use these days.

      Of course since Windows is "out of favor" here, one does not necessarily mention that Microsoft's "Security Essentials" is easily as good as most commercial Windows anti-malware packages, and much more "light weight". And free. And yes, everyone knows that Microsoft purchased the original technology (so what?) ...

      MSE is good for what it is and what it does, I first tried it after reading unanimous praise of it here on Slashdot. It's the only AV I've ever seen that does not conspicuously cause the system to become slow, unstable and/or quirky.

      I am feeling smug about this and is not about Microsoft or Windows itself, I just simply could not understand how a professional sysadmin could ever be in a position where they must run anti-virus on a server, which seems to be common practice amongst Windows admins.

      Antivirus is for checking that executables and libraries are free of malicious code. I just cannot possibly fathom why an executable or library could be running on a server if nobody had checked it beforehand. A good admin should scan and monitor tools that come from untrusted sources before putting it on a live server. A great admin should scan and monitor tools, even if they're from trusted sources before putting it on a live server. This is basic stuff and is why almost all servers are infected through network bugs, which can be easily prevented by keeping services up to date and non-essential services shut down or at least firewalled off.

      Why then do you need an Anti-Virus? It won't protect your services from buffer overflows or other infection vectors, it won't protect you from new rootkits unless it has wicked-sick heuristic analysis and you get lucky. So what does it guard against? Maybe someone using a zero-day attack vector and installing an old rootkit?

      So for a sense of security against unknown threats, you give an autonomous, externally controlled process, that is by design almost impossible to analyse, unfettered administrator access to your entire system. Now this happens, I feel smug.

      • by DarkOx (621550)

        ntivirus is for checking that executables and libraries are free of malicious code. I just cannot possibly fathom why an executable or library could be running on a server if nobody had checked it beforehand.

        .
        You are making assumptions about things you don't and can't know. Is the a vulnerability in you web application that lets someone put a file? Could they then get some server side processing to happen on that file with another crafted URL?

        As much as we try to prevent them these things happen. Unless you as an admin are also auditing the source code to every server process you run; its entirely possible your box will be pwnd due to the mistakes of others.

        To say nothing of your own mistakes. AV on servers

    • Where can I get ' Microsoft Security Essentials ` for Linux?
    • by hairyfeet (841228)

      As someone who actually has to do the cleaning when viruses get in there is a serious problem with MSE, which just for full disclosure I use myself on my netbook and gamer box, and it is thus: It works well IF and ONLY IF you are already using best practices and not going anyplace risky.

      Now the reason why is actually VERY simple, and its why MSE is so much lower resource than other AVs out there...its not really an AV in the traditional sense at all. You see it was originally Giant AntiSpy which like most a

    • by kwerle (39371)

      Right, and it's what I use and recommend.

      Which begs the question: why do I have to install it? Why doesn't it ship with?

      I mean, sure, someone is in bed with the various AV vendors. But when you ship an OS that is for use by joe-users, you really ought to keep it clean.

      Whatever. I find it frustrating.

    • ...is all I use these days.

      Of course since Windows is "out of favor" here, one does not necessarily mention that Microsoft's "Security Essentials" is easily as good as most commercial Windows anti-malware packages, and much more "light weight". And free.

      Never used Microsoft's "Security Essentials" only because of back door issues. While I know of none,
      I just don't trust MS and some programs I run MS would strongly object to (like linux :)

      For the record I use ESET aka NOD32.

      NOD32 is set to alert me to a problem so I can decide what to do about it not the program. Default
      is to not only quarantine it, but encrypt it as well. At least NOD32 lets me have the option to change that,
      many programs don't feel the user has the ability to know a good file from a bad

    • One, two, three, four.
      I declare a shill war.

    • Everyone keeps saying that MSE is lightweight and doesn't bog down your computer, but it seems that more and more often recently, I've seen it max the CPU for a minute or more, for no apparent reason. This is on many different machines with many different configurations, so it's not a single data point,either.

    • by jimicus (737525)

      Never mind MSE - which is only on a subset of Windows computers.

      Microsoft have recommended uninstalling a core Windows 7 patch in the last week or so: http://support.microsoft.com/kb/2839011 [microsoft.com]

      Face it, anything that involves changing how a computer operates - regardless of whether the process for making those changes is automated or manual - introduces risk. You just have to decide how big the risk is, weighed against the alternative.

  • by girlintraining (1395911) on Thursday April 18, 2013 @12:29AM (#43479737)

    "I don't understand... it worked fine in the lab."

  • Just was in the process of downloading a beta client for their new online backup system to fiddle around with on a virtual machine (it is similar to Mozy/Carbonite.)

  • Always use Genuine Microsoft Products

  • Doh! (Score:4, Insightful)

    by All_One_Mind (945389) on Thursday April 18, 2013 @12:30AM (#43479745) Homepage Journal
    For once I'm happy that I'm too lazy to regularly update programs like that.
  • by tftp (111690) on Thursday April 18, 2013 @12:32AM (#43479755) Homepage

    How many viruses your antivirus caught recently? How many CPU cycles the same antivirus burned through as you were opening files on your computer?

    Maybe I'm doing something wrong, but I haven't seen a virus in a decade. The majority of successful attacks are based on social engineering and on 0-day exploits of vulnerable code. An antivirus is not such a great help here. But antivirus companies are sitting pretty because the audience is conditioned that any PC must have an antivirus.

    • by Anonymous Coward on Thursday April 18, 2013 @12:42AM (#43479793)

      I've yet to see an AV that actually can deal with browser add-on attacks.

      The only thing that might help is Malwarebytes because it blocks by IP address.

      If you want protection, use an ad blocker. Ad servers seem to be one of the chief causes, if not the top infection vector these days.

      • by Arker (91948)

        Mbam is one of the best on the field today.

        The field is pretty crappy though.

        To understand the situation you really have to go back to the 80s. Antivirus scanners were just starting. Some of us were pointing out the problems with it. Some of us even made non-scanner AV systems that worked. Give me a DOS6 system and I can give you a very effective automatic defense system (though it would naturally take some time, given how many of the details I have forgotten between then and now.) Windows versions 3 and la

        • by Fjandr (66656)

          I'm not sure why people are enamored with Malwarebytes. I honestly have not seen it fix or prevent anything, and I've tried it a number of times because of the praise it receives. I've fixed a lot of machines that had it installed, and have never seen it do anything useful.

    • 1 in 20 (Score:4, Insightful)

      by tuppe666 (904118) on Thursday April 18, 2013 @12:44AM (#43479795)

      Maybe I'm doing something wrong, but I haven't seen a virus in a decade.

      ...or maybe as http://eugene.kaspersky.com/2013/03/25/one-in-twenty-is-the-sad-truth/ [kaspersky.com] "Even those who care nothing for their health still get sick – it’s just that the infection goes undiagnosed" as much as you may find it comforting blaming users, 1 in 20 infected machines implies there is something wrong. Its no wonder users are not buying PC's anymore.

    • So, basically an antivirus program is just like the TSA, catches nothing and slows down the process..

    • by woboyle (1044168)
      I only run Windows in a Linux virtual machine. If it gets a virus, I just revert to the last snapshot. That said, I do run ClamWin (ClamAV for Windows), but it only runs scans when I want, such as when I think that something is trying to get in my "pants". I do AV cleansing for clients, but I use ClamAV and 2 other professional-level scanners on a Linux system. I connect the infected drive to my linux system using a docking bay, make a bit-image backup of the drive and file systems, and then scan the file s
      • I was right with you until:

        Then I clean the system.

        ... Malware authors typically snag a new piece of malware then modify it, malware typically installs other malware also potentially mutated. You can't clean the system. You just gave them back a machine you weren't sure was actually clean. What's to say you just didn't find one of the many quieter variants?

        Just to be perfectly clear: You CAN NOT Clean malwale. You can restore to a known good state with a VM. Otherwise: Unless you were watching that thing instruction by inst

    • How many viruses your antivirus caught recently? How many CPU cycles the same antivirus burned through as you were opening files on your computer?

      Maybe I'm doing something wrong, but I haven't seen a virus in a decade. The majority of successful attacks are based on social engineering and on 0-day exploits of vulnerable code. An antivirus is not such a great help here. But antivirus companies are sitting pretty because the audience is conditioned that any PC must have an antivirus.

      Either you're not exploring the web, or unaware of any infections (or you practice safe cyber-sailing).

      While an anti-virus solution won't help with 0-day exploits, it may eventually (or should) indicate some sort of problem. You might not catch it on day 1, but if you've missed all the other signs of an infection (or aren't watching for them), then an AV install that won't update is an EXCELLENT way to detect a problem.

      • by tftp (111690)

        Either you're not exploring the web, or unaware of any infections (or you practice safe cyber-sailing).

        I must admit that IRL I also do not explore sewers, and don't go after midnight into a bad part of town, and I don't instigate bar brawls, and I don't bother sleeping dogs. You might classify me as "cautious."

        As far as being aware of possible infections... I have MS AV running; it is a low maintenance thing, so I let it be. It's not great, but what is? A skilled, targeted intrusion, such as a stealth

    • by wvmarle (1070040)

      As you apparently don't run any anti-virus or other anti-malware software, I'm not very surprised you don't see any of the possibly dozens of viruses that have infected your computer.

  • Why on earth would someone update software like this on production systems, instead of testing it in a lab environment first?

    Anyone that knocked 80% of our servers offline by applying this patch would be packaged out the next day.

    • Re:Production (Score:5, Informative)

      by gweihir (88907) on Thursday April 18, 2013 @12:47AM (#43479805)

      AV software (or rather its definition files) has to be updated very fast if it is to have any value at all. You cannot qualify it for production, that takes too long. This is one reason the whole concept is fundamentally flawed, because it is still too slow.

      • AV software (or rather its definition files) has to be updated very fast if it is to have any value at all. You cannot qualify it for production, that takes too long. This is one reason the whole concept is fundamentally flawed, because it is still too slow.

        ... Unless you're running an unpatched/exposed version of something, but aren't exposed on day 1 (or 0, as it were).

        • by Arker (91948)

          Exactly, signature antivirus only protects those who use it properly (most dont) AND luck out by not being among the first exposed to the new mutation of the day. Heuristic scans usually wind up with way too many false positives to be useful. These are just vain attempts to patch over an insecure core.

          Securing the core would make everyone from marketing and a good portion of engineering extraordinarily unhappy by ruling out cool junk they would love to see and sell. You cant even sell that notion in linux l

      • by 1s44c (552956)

        It's wrong because it assumes everything is good unless it's on the AV naughty list, hence the panic to distribute new naughty lists so quickly.

        The whole anti-virus and anti-malware thing is a product of an OS that is incorrectly designed.

    • Why on earth would someone update software like this on production systems, instead of testing it in a lab environment first?

      Because they assumed Malwarebytes had done that already.

  • There is no way to prevent these things from happening. It is just not possible to test them on all the individual versions of a platform. On the protection side, AV only works against older threats, it is basically useless against new ones. There is no replacement for careful users and good software engineering.

    • by Spikeles (972972) on Thursday April 18, 2013 @01:40AM (#43479985)

      There is no way to prevent these things from happening

      Sure there is. Kaspersky Anti-Virus Security Center has a Update Verification [kaspersky.com] module built in, that allows a sysadmin to install the update to a known-clean test group and then run a virus scan BEFORE the update is applied to the rest of the machines. If the scan fails(ie, finds anything), the update is aborted and an email is sent to the admin. If Malwarebytes had that kind of thing(or if it did and the sysadmins actually used it), this wouldn't even be an issue.

  • Servers??? (Score:3, Interesting)

    by Holi (250190) on Thursday April 18, 2013 @12:50AM (#43479821)

    What the hell are you doing running malwarebytes on your servers? Why would you need that software on a server, most of the malware it finds is installed from desktop use.

  • by mevets (322601) on Thursday April 18, 2013 @12:57AM (#43479843)

    It identified the malware, disabled it, and everyone gets upset...
    no pleasing some people

  • Rhetorical questions: based on the large-surface high-impact outcome, wouldn't this qualify as a blatant case of cyber-terrorism or cyber-war? Now, where's that nuclear strike from NATO [slashdot.org]?

    (my point: before trying to stop vulnerability exploitation by moronic laws [slashdot.org] or DCMA-export treaties [tppinfo.org], wouldn't it pay better to clean your own yard? You know? It may be beneficial no matter who if the "aggressor" is a script-kiddie or North Korea [bbc.co.uk].
    But... who am I kidding? Doing this require some competence and thus would be t

  • ... depends basically on what amounts not much more than a grep tool.

    False positives.

  • A few points... (Score:5, Interesting)

    by waspleg (316038) on Thursday April 18, 2013 @04:33AM (#43480555) Journal

    1.) I've been using MS Security Essentials for YEARS without issue and have it running on many machines also without issue, not it does not catch EVERYTHING; but nothing does. It does a pretty damn good job for something ad-free, shitware-bundle free. Other than the occasional annoying "OMG YOU HAVEN'T SCANNED ANYTHING!@#!@ orange flagged monopoly house ! warning, is pretty unobtrusive.

    2.) All Windows versions prior to 8 could also use Windows Defender in addition, if you want to, but they've been rolled together under the Windows Defender name and are included by default in Windows 8.

    3.) Microsoft also has a Malwarebytes-like scanner called Safety Scanner although it auto-expires after 10 days and has to be reinstalled for subsequent use; no idea why.

    4.) 0-day exploits by definition would be more or less impossible to defend against, wtf is the problem? I'm no MS fanboy, but the hate here is unwarranted, they're basically risking massive lawsuits against them again for anti-trust by even doing this and frankly it's about fucking time they should have had all of these tools available from its inception.

    5.) Malwarebytes has gone from a must-have awesome malware scanner to total shit adware in the typical bait-and-switch style business model of the day which goes something like a.) build something awesome b.) give it away for free c.) change to paid model with your own bundled malware and bullshit once it gets popular d.) crash and burn e.) laugh all the way to the bank.

    Where I work uses Sophos, I would say it's far worse (and used more as an attempt at draconian control than really A/V, and does next to nothing for malware, updates fail constantly, etc), and I've actively advised people to not use Macfee and Norton for a very long time because of all their dumb bullshit problems. Clamwin is still pretty terrible and ridiculously slow, after all these years. I think the only one I've never used at all is Kapspersky, or whatever.

    $.02
     

    • by 1s44c (552956)

      Actually it is possible to defend against most zero day exploits. Good design prevents most of them happening in the first place and security in layers reduces the risk if they do exist. Firewalling windows machines as much as possible is essential if you need to use these things. And use a real firewall not the windows software nonsense.

      I use Kaspersky on some systems and it works well. Give that one a try. I think they do free trials.

  • Malwarebytes (Score:4, Insightful)

    by 1s44c (552956) on Thursday April 18, 2013 @05:13AM (#43480683)

    The clue is in the name.

  • I've been using them for years and I've never had a problem (in fact they've saved my ass on several occasions); it was just one mistake so I think I'm going to keep using them.
  • The best solution for windows is to start it as a fresh VM at each reboot. No problem of malware or virus or performance degradation. I can reboot windows without stopping my work.
  • My first and only story on /. was about when this happened before. Last time around, Malwarebytes removed atapi.sys from affected computers, leaving them unable to boot.

"A mind is a terrible thing to have leaking out your ears." -- The League of Sadistic Telepaths

Working...