Forgot your password?
typodupeerror
Security Bug IT

McAfee Retracts Lowball Bug Damage Estimate 233

Posted by kdawson
from the had-our-fingers-crossed dept.
bennyboy64 writes "McAfee has changed its official response [warning: interstitial] on how many enterprise customers were affected by a bug that caused havoc on computers globally. It originally stated the bug affected 'less than half of 1 per cent' of enterprise customers. Now McAfee's blog states it was a 'small percentage' of enterprise customers. ZDNet is running a poll and opinion piece on whether McAfee should compensate customers. ZDNet notes a supermarket giant in Australia that had to close down its stores as they were affected by the bug, causing a loss of thousands of dollars."
This discussion has been archived. No new comments can be posted.

McAfee Retracts Lowball Bug Damage Estimate

Comments Filter:
  • XP SP3 (Score:4, Insightful)

    by Enderandrew (866215) <enderandrew.gmail@com> on Friday April 23, 2010 @10:27AM (#31955022) Homepage Journal

    I thought this affected anyone running XP SP3, which I expect would be a majority of enterprise desktops, not less than half of one percent.

    • Re:XP SP3 (Score:5, Insightful)

      by SharpFang (651121) on Friday April 23, 2010 @10:29AM (#31955060) Homepage Journal

      I guess less than half of 1% of all corporate customers are customers of McAffee.
      The right wording is everything.

      • by poetmatt (793785)

        yeah, the media spin is strong with mcafee.

        Reality? It affected everyone who has automatic updates on mcafee for enterprise, which roughly translates to a large majority of enterprise customers. Usually from a security perspective it's seen as bad form to not have updates available as soon as possible.

        It also shows that mcafee's quality control is nothing short of crap. It's known that viruses do rename as svchost sometimes, but clearly they didn't test the heuristics here.

      • I would guess there are more than that because of previous licensing. Luckily their licensing ran out on us and we switched to Norton since McAfee hasn't really done much since 2003. There enterprise stuff has really sucked for a while now but we had to wait to get out of the deal with them because of "you know" the economy.

        • I really wouldn't trust Norton any more then McAfee.

          Honestly - I don't know what the right answer for a corporate entity is... There is just something really scummy about both companies that I don't like.

          • Re: (Score:3, Informative)

            by Enderandrew (866215)

            Microsoft Forefront is what I'd suggest.

            • I really like MS Security essentials... I hate to say it.. but I actully do trust Microsoft much more then McAfee and Symantec. I would try this out in a heartbeat.
          • by thsths (31372)

            > Honestly - I don't know what the right answer for a corporate entity is...

            Sophos is another good choice. But really any choice is better than Norton or McAfee. Avoid these at all costs.

    • Well, it depends. How many have their computers set to pull updates hourly? If you pulled the updates daily, and it was released an hour after you checked, you were fine (considering they pulled it the same day). So the only computers affected were those that polled in the several hour window that the update was available (Something like 8 hours IIRC). And that's not to mention those configurations that are set to pull updates weekly or more.
      • by kiehlster (844523)
        You should also add to this the statistic of how many corporations use their own distribution server (middleman). Even if clients poll daily, the corporation as a whole may only deliver updates weekly or may stagger updates to ensure they are tested in the wild before pushing them out to corporate clients.
        • Re: (Score:3, Informative)

          by Jazz-Masta (240659)

          You should also add to this the statistic of how many corporations use their own distribution server (middleman). Even if clients poll daily, the corporation as a whole may only deliver updates weekly or may stagger updates to ensure they are tested in the wild before pushing them out to corporate clients.

          Not only this, but many Administrators manually review virus' before they are cleaned. I have caught a few false positives by doing manual checks.

    • Well, one condition - that the v8.7 McAfee app scanned the svchost.exe file of a WinXPsp3 machine.

      Which could happen under three situations:

      1. You manually launched a scan.
      or
      2. A scheduled scan launched.
      or
      3. A setting in your policy said "scan processes on enable".

      • In most enterprise environments McAfee is going to have real time protection against running processes. Can you point me to an enterprise environment where this wouldn't be the case?

        • by khasim (1285)

          In most enterprise environments McAfee is going to have real time protection against running processes.

          It is "real time protection" even if that setting is set to "off".

          McAfee's documentation specifically mentions turning it off because there is a high processor utilization bug still in it. Although you'd need to read the "read me" file that came with the patches.

          Other than that, unless you choose the highest security setting, it is off by default in a BRAND NEW VANILLA install. But not if you had upgraded

          • by thsths (31372)

            > McAfee's documentation specifically mentions turning it off because there is a high processor utilization bug still in it. Although you'd need to read the "read me" file that came with the patches.

            And stupid me thought that high processor utilization is a "feature" of McAfee. Seriously, if it is bug, why has it been there for years if not decades?

    • Re: (Score:3, Interesting)

      by GIL_Dude (850471)
      It really depends on the intersection of folks running McAfee along with SP 3 in the enterprise. My company is just finishing a migration to Vista, but we still do have about 15,000 Windows XP SP3 desktops (not done deploying yet). However, late last year, I was at a MS Global Accounts meeting (35 very large companies) and NONE of the rest of them had deployed SP 3 for their XP machines. They were all on SP 2 and were harping on Microsoft about the end of support for SP 2 that was fast approaching. None of
      • by swb (14022)

        None of them wanted to deploy SP 3. It was flabbergasting to me, but they just didn't want to do it.

        Some fucktard in a suit gets told that they don't care about problems caused by not running SP3, running SP3 requires a bunch of money to get spent and if he spends it he doesn't get a new BMW 7 series this year.

        Really, so many of these decisions have nothing to do with rationality. At some high level it comes down to some guy in a suit angling for a new car, a new house or some other luxury/status symbol.

    • At my work we run XPSP3 and McAfee, had no problems here.

      @WithinRafael on Twitter (from www.withinwindows.com) was trying to reproduce it and had problems, I think he recently succeeded but hasn't provided details yet.

      • He tried to reproduce it and had problems? The summary of the problem made it seem like all svchost.exe's would get deleted no matter what.

        I wonder what sort of specific conditions had to be met? Not that I like coming to the defense of McAfee... But has this been overblown?

        • He tried to reproduce it and had problems? The summary of the problem made it seem like all svchost.exe's would get deleted no matter what.

          I wonder what sort of specific conditions had to be met? Not that I like coming to the defense of McAfee... But has this been overblown?

          We were hit by this but I called the guy who manages the AV server and told him to halt any updates and roll back to 5957. Only about 15 systems were hit with it, but none of them had SVCHOST deleted. I was able to isolate one and it was fine since we didn't have the "scan process" enabled. Here is an e-mail I sent to my department:

          1. It was on 5958, but everything was running fine.
          2. Since I knew there was a fix, I ran an on-demand scan.
          3. McAfee picked up SVCHOST.EXE as a virus, and it tried to delete it but the clean failed.
          4. Since the clean failed, all I had to do was manually run SVCHOST.EXE from the command line, force an update by right-clicking on the McAfee icon in the systray, and reboot. I ran another memory scan and there were no red flags.

          And for this:

          I wonder what sort of specific conditions had to be met? Not that I like coming to the defense of McAfee... But has this been overblown?

          Specific conditions had to be met, but they were broad. The following were necessary:
          - Windows XP SP3
          - Real-time Scanning Enabled
          - Definitions version 5958

          • To rephrase, "a LOT worse for us." A buddy of mine had to get their entire IT department to go around by hand and fix every computer by hand.
    • by proxima (165692)

      I thought this affected anyone running XP SP3, which I expect would be a majority of enterprise desktops, not less than half of one percent.

      You had to be running versions 8.7 or 8.9 it seems to be affected. 8.0 or 8.5 did not exhibit this problem, even if the virus definitions were updated to 5958.

      It wouldn't surprise me if the enterprise rollouts of McAfee often used 8.5 (released in Nov 2006) rather than 8.7 (released in Sep 2008) or newer.

    • Presumably at least a few enterprise customers have enough brains to internally test updates before rolling them out. I expect McAfee doesn't consider those customers "affected".

    • by Piranhaa (672441)

      Everyone that received the patch running XP SP3, yes. However, where I work, they download the patches in the morning and deploy them later on in the evening. So yes, there is a window of attack there, but it saved us from having to go through every SP3 machine and copying the deleted OS file. Basically, everyone else that gets the patches instantly are 'our' guinea pigs.

    • It also affected W2K3 servers.

  • by ircmaxell (1117387) on Friday April 23, 2010 @10:27AM (#31955024) Homepage

    ZDNet notes a supermarket giant in Australia that had to close down its stores as they were affected by the bug, causing a loss of thousands of dollars.

    A chain of supermarkets close down, and they only lose thousands

    of dollars? Really? I would expect that figure to be a lot higher than that for a single store... Think about all the fresh produce that'll go bad (that have daily deliveries). Think of the power usage (lights, refrigerators). And that's assuming that they aren't paying any of their employees while the store is closed. I'd imagine the loss would be on the order of tens of thousands of dollars per store. Not thousands of dollars across all of the stores...

    • Re:Really? (Score:5, Funny)

      by pinkj (521155) on Friday April 23, 2010 @10:45AM (#31955296)
      Maybe Australia only has one big grocery store somewhere in the Outback. Kinda of like what we have in Canada except it's a giant igloo in northern Toronto.
    • by kiehlster (844523)
      I would think the same, but it could be a discount supermarket with really low profit margins on dirt-cheap products from second-rate suppliers. We have a chain like that in our area where they leave out the produce until it gets moldy and then offer a replacement guarantee. So if you're 5-day old fruit turns moldy on you, you can return it, but they don't have to toss out as much because people tend to use the fruit within a day or two of purchase. If this was a reputable supermarket, I could see shorte
      • Re: (Score:3, Interesting)

        by Cimexus (1355033)

        Nah - this is Coles. That'd be one of the "big two" Australian grocery retailers, with thousands of stores nationwide. I expect that 'loss of thousands of dollars' was many, many thousands (either that or it only affected a very small number of stores for a very small time before getting fixed).

        Actually I used to work at Coles (it was my first job!). Our store was the smallest one in the state but still had revenue of ~$300,000 a day...

    • by eggoeater (704775)
      Agreed. And that's just the immediate cost. When things like this happen, stores/businesses lose loyal customers to competitors and it takes months to recover.
      And what about the IT costs? I guarantee you, there is now an effort underway in all major businesses to (1) test new anti-virus patches before rolling them out, (2) re-review all anti-virus software being used, (3) developing and testing mitigation plans for another failure. All of this is VERY expensive.
      Here's another example: Airlines shut d
    • At least one of our customers were affected as they run our point of sale software on XP Pro SP3 and used McAffee as their anti-virus. That was the IT environment they chose, we told them we prefer OSX as our first choice/Linux as second choice, but they already had a previous POS solution deployed on Windows.

      They've requested price quotes on the OSX and Linux hardware solutions.

  • by khasim (1285) <brandioch.conner@gmail.com> on Friday April 23, 2010 @10:29AM (#31955050)

    ... why they didn't test the new dat file against Windows system files.

    Seriously, we pay them a LOT of money for their product licenses and they cannot even test against known system files?

  • I wonder (Score:3, Interesting)

    by mr_da3m0n (887821) on Friday April 23, 2010 @10:32AM (#31955090) Homepage

    ...If McAfee has a clause in their EULA somewhere that limits their responsibility, and should that be the case, if it is legally enforcable.

    Maybe someone with access to said EULA could look it up?

    Microsoft once pushed their accountability as a selling point for the Windows Server platform against Linux, if I recall well -- however their maximum responsibility was something like 50$. I wonder what is McAfee's stance in this regard.

    • by green1 (322787)

      By the time you are dealing with large enterprise customers, you aren't dealing in EULAs anymore, you're dealing in negotiated contracts where the legal department of each company goes over each and every clause in the contract.
      I was talking with some of our IT folks as this unfolded (as my work machine was one of the ones affected) apparently after we were bitten badly by a vendor bug a few years ago, we re-negotiated with most of our software vendors. Our contracts now include penalty clauses for this sor

  • Everything here is windows xp sp3 with McAfee installed.

    Fortunately for us, all software updates are filtered through and managed by an internal server due to security restrictions on some of the work we do for the government.

    • all software updates are filtered through and managed by an internal server due to security restrictions on some of the work we do for the government.

      And this is a perfect example of why an internal server to distribute updates is a Good Thing(TM). Hey, the government got something right!

      • by chill (34294)

        Hey, the government got something right!

        Whoa there, pardner! Before jumping to any wild conclusions, re-read what he said.

        ...on some of the work we do for the government.

        That most likely means contractor, not actual government employee.

        The gov't didn't do something right. The world is not going to end. Moped Jesus was not spotted on I-55 heading west.

        • Yes, but it is the government who put that stipulation in for the contractor. So I am still maintaining they did something right. Whether or not the contractor actually works for the government or is just contracted is irrelevant. The stipulation is there and is there becasue of the government.
  • Necessary Evil (Score:2, Interesting)

    by RayRuest (1417225)
    It could only effect that few if the policies were set up update infrequently (ever few days or so). My policies are set to check for updates and push them frequently, so I got bitten. I have less than 100 desktops but am a 1 person shop. 4 hours of sneaker net repairs and corporate downtime. Thanks McAfee. There was at least 1 hospital in the area that had to resort to turning non-critical patients away. Don't these things get testing before release? These products are a necessary evil... they don't
  • Heck I was at a small IT security trade event yesterday and like a quarter of the attendees had to cancel because they were dealing with the aftermath...

    McAfee had almost a 50% corporate AV market share, and nearly all of those companies still run many XP SP3 boxes. If 10% pulled the DAT before it was yanked, that's a metric buttload of machines...

  • Is that it would only take 1 oil and gas company who usually handles Million Dollar deals. Lets see.
    International Corporation... Lets say 3000+ Employees... lets say just half the company goes down. Rule of thumb is 1 IT guy for every 100 computers (but we all know thats in a perfect world).
    So, the simplest way to get out of downtime is to go into safe mode and disable the Antivirus, right? Lets say it takes on average 5 minutes to walk to each machine and preform the steps. 500 minutes, or 8.3repeating hou

    • Chances are this will put McAfee out of business for more than a day, so I guess it all balances out.

  • I've read a few interviewed accounts where the story was much like this:

    We applied the updates, and rebooted, then I went on to kick off the others. When I went back to the first couple of servers, I noticed they had rebooted again... then I knew something was wrong.

    I know things can't be 100% perfect in an IT world, and yes, virus definitions can be touchy when sometimes zero-day shit can really cause havoc, but I, myself, have of test boxen on my network that I test all patches/updates/virus definitions on for *NIX and Windows boxen. It's not perfect, because to test and interrogate everything is impossible, but I don't apply things blindly. And yes, I've had a few fallout where the package/pat

    • by X0563511 (793323) on Friday April 23, 2010 @10:55AM (#31955468) Homepage Journal

      I know assumptions are bad, but is it really that big a stretch to assume the vendor tests their updates on their supported platforms?

      It's not like these were weird corner-cases.

    • by alen (225700)

      i've been using Winders since the mid 1990's along with AV software. I have never seen an issue where a definition update has caused something like this. i've seen plenty of times where you can't run an old version on a new OS or issues with games or some software. but letting something out like this into the wild just shows that there was no testing done just to make sure it's OK

  • by wvmarle (1070040) on Friday April 23, 2010 @10:44AM (#31955288)

    I feel sorry for that super market chain but: wtf is AV doing on a POS computer?

    POS should be a dedicated computer, running one and only one application (the POS software), on a thoroughly shielded LAN, talking to only a centralised server (or small network of servers if one is not enough) that collects the sales data and distributes prices etc. That server should itself be connected only to the POS network and a corporate LAN. In other words: no direct access out of the Internet, no web browsing, no local storage of any data files, no downloading, nothing that could have the most remote risk of a virus.

    Or am I missing something here?

    • by ifrag (984323) on Friday April 23, 2010 @11:00AM (#31955566)

      Or am I missing something here?

      That it was in Australia?

    • by Anonymous Coward on Friday April 23, 2010 @11:03AM (#31955612)

      wtf is AV doing on a POS computer?

      This setup also seems somewhat redundant, since McAfee's AV itself is a POS.

    • McAfee must have had a really good sales guy to convince a Project manager that the POS machines needed AV, either that or who ever developed the POS machines didn't decide to secure them with Enhanced Write Filter, SteadyState, DeepFreeze or some other disk write protection so every time the machine is rebooted it loses all its write cache.

      Even though it is Windows, there is absolutely no need for AV when the application is so limited.

      • by knarfling (735361) on Friday April 23, 2010 @11:56AM (#31956424) Journal

        Even though it is Windows, there is absolutely no technical need for AV when the application is so limited.

        Fixed that. I am afraid that the Payment Card Industry (PCI) differs from your opinion.* In their infinite wisdom**, PCI has decreed that ALL computers need to be running AV. After, all, if it is good for the desktop, it must be good for the servers, right? And since a virus can be spread from anywhere to anywhere, all computers need to have their own protection.

        I know it seems silly, but many of the PCI Audit Drones actually believe this. I spent hours trying to convince an auditor that we did not need AV on a Linux server that cannot accept email and has no internet connection. If the PCI Audit Drone finds a computer without AV, you fail the PCI Audit. If you fail the Audit, you get marked as failing on a public web site. If you fail enough times, you lose your ability to accept credit cards. So the need to have AV on a POS is there, it is just not a technical need.

        *Reality
        **For very, very small values of infinite

    • Re: (Score:2, Insightful)

      by EMG at MU (1194965)
      I agree.
      However, when you have 200,000+ POS machines, management wants an AV.
      I hate McAfee, I hate using a AV instead of isolating a machine from removable media and the Internet. I hate spending money on AV when we could use it on something else. But when a franchise manager on the other side of the world lets one of his employees use the wifi or a printer or something, I'm glad there's an AV to protect my ass. Even though there shouldn't be a way the POS machines get a virus, the AV is kind of like ca
      • by wvmarle (1070040)

        the AV is kind of like car insurance: It protects you from accidents

        Since when does insurance protect you from accidents? It only compensates you when an accident happened already. If you want to have a car analogy then you should compare AV with seat belts or air bags, that are prevention measures.

        • Air bags and seat belts don't protect you from accidents either. But, I think they are a good analogy for AV software. You still have an accident and it still hurts, but you are less hurt and might survive because of it.
    • by Artifakt (700173)

      Most small businesses that are service related have at least one Point Of Sale machine up front at their physical store, but the person operating it is also the person who makes appointments, so they just about have to be able to bring up a scheduler and appointment manager. A separate terminal for appointments is a serious cost, as would be keeping separate people to operate it, or training across skill sets (your cosmetologist or hair stylist or auto mechanic now needs to be trained to schedule appointmen

    • Re: (Score:2, Interesting)

      by Scyth3 (988321)
      Typically the POS desktops are talking directly to a server in the backroom. The server in the backroom is typically where a manager will check their emails (via Outlook), take training via a web site, etc. and it's also where the database for the POS client desktops is stored. Every night that small store server submits the data to a main server at the "home base". So, if the virus scan is on the server (typically is), and the machine goes down, then the business is effectively closed. It's not that th
    • by c (8461)

      > Or am I missing something here?

      Slavish adherence to corporate IT policies which require AV software on any system which can run it?

      c.

    • You're missing nothing except one minor point: no POS system - or anything else in the chain - should be running Windows. This should be a non-issue. My advice to the Australian grocery chain is to fire whomever in the IT department thought this was a reasonable idea.

    • Re: (Score:3, Insightful)

      by Locutus (9039)
      and why does a POS computer have an internet connection to get the updates? It reminds me of the story of how a bunch of trains had no signal systems because the computers controlling the railway signals were running Windows, connected to a LAN, and got infected with a virus and stopped operating the signals. I guess with admins, you get what you pay for and maybe those MCSE certs are worthless.

      LoB
    • It's required by PCI-DSS. Anything that is touching Credit Card data has to be running AV. Our e-commerce servers run on FreeBSD. Guess what, they're running ClamAV. Not because there are viruses for FreeBSD, but it's a PCI requirement.

    • by Bert64 (520050)

      It is generally accepted practice that windows systems _require_ av, wether it does much good or not is highly debatable (i do a lot of incident response work - ie identifying the source of a breakin, and every system that i get to investigate has some kind of av installed slowing it down)... Infact, i have often had people complain about linux or mac systems without av installed. It's very hard to fight against "standard practices" even when those practices are blatantly flawed.

      Ideally such devices wouldn'

  • by goffster (1104287) on Friday April 23, 2010 @10:46AM (#31955308)

    McAfee or being part of a botnet?

  • by onyxruby (118189) <onyxruby&comcast,net> on Friday April 23, 2010 @10:50AM (#31955380)

    First, McAfee blew this big time, that such a bug made it to production shows a complete breakdown in their internal processes. XP with SP3 is the number one OS combination in enterprise environments, and should have been the first thing that they tested on. Without doubt McAfee has liability on this and needs to get aggressive about damage control with clients.

    That being said, every one of these clients that was hit by this is just as guilty as McAfee is! They are in no better shape and those responsible need to be going management review for their failure. Enterprise Management 101 - nothing goes into production that has not been tested in a lab for pre-pilot and a small group of production computers for pilot! This is as basic as enterprise management gets. Every single environment that was taken down by this shows professional incompetence by their requisite IT departments.

    The only question is if it is the fault of management for failing to allow the budget and support needed for a lab for testing or if it is the fault of the IT staffer who never tested things as they should. This is without doubt one of the most public examples of IT incompetence to make the news in years. This is a case of sheer and utter incompetence by every affected party and no pity should be given. If pity were to be given, give it to the poor desktop techs that have to go around making apologies and manual fixes for everything.

  • by ProdigyPuNk (614140) on Friday April 23, 2010 @10:50AM (#31955386) Journal
    A buddy of mine is in IT at a college in the area. This affected almost all of their computers. Although it's harder to put a dollar figure on, the students and professors were NOT happy when all of the computer labs on campus went down, along with a "server" or two. Ever seen professors gets mad ? Now imagine your an IT guy and the professors can't access their online grade books that you pushed them into using. I really think McAfee is going to have a big problem on it's hands come contract renewal time. Pissed off IT people have long memories!
  • We use Sonicwall's security services, their anti-virus is a crippled version of Mcafee business. And we've been hit hard: Machine where going down but WITHOUT any explanation or any warning messages (this version is silent to the user) and since svchost was killed, no chance of getting in the event monitor or using any tools, it took me couple of hour to figure it was the AV. I am sure they "forgot" to add all those third party security solution who rebrand Mcafee solutions. What is making me mad is the way
  • Oblig. xkcd (Score:5, Insightful)

    by wvmarle (1070040) on Friday April 23, 2010 @10:53AM (#31955438)
    Quite apt, even though not POS: http://xkcd.com/463/ [xkcd.com].
  • Damage Limitation (Score:3, Informative)

    by MrNemesis (587188) on Friday April 23, 2010 @11:11AM (#31955726) Homepage Journal

    "McAfee Interwebs Secrutiny has detected that your outgoing mail to customerservices@mcafee.com, subject "You f**king idiotic t**tballs of a son of a ****** in the ******** with a hatstand!!!!" has been detected as Offensive Spam and will be deleted. Thank you for Trusting in McAfee! [TM]"

    On a more serious note, I ran into a few small shops that were badly hit, but most of the people I know who work in the enterprise have a time delay before the updates hit the machines, which is usually a hangover from the last time $av_vendor bollocksed up an update.

    Personally, I'm still a believer in most AV's being worse that the viruses themselves, and don't run any on my windows boxes - I don't think I've used a single one that hasn't fucked up at some point. Most of my colleagues feel the same way (and, IMHO, by the time it's hit your filesystem and you have that 20% chance of the AV detecting it, it's already too late anyway) and the only reason we run it at work is because of compliance issues... that and the majority of machines being a poorly patched IE6. Yay!

  • "ZDNet is running a poll and opinion piece on whether McAfee should compensate customers."

    Poll? Opinion piece??? This is fucking America. Spare me the nonsense, show me the lawyers.
  • by Atreide (16473) on Friday April 23, 2010 @11:16AM (#31955818)

    we have 11K computers

    only XP SP3 computers were impacted
    whether running Virus Scan 8.7 or 8.5

    but in fact less than 100 computers were impacted,
    1% compared to our total

    one thing that helped
    was employees had started to leave after work when update propagated
    and they shutdown computer when they leave

    it could have been a nightmare
    we were very lucky

  • by Atrox666 (957601) on Friday April 23, 2010 @11:48AM (#31956286)

    When was the last virus outbreak that caused this much damage?

  • All that time the computer weren't running windows. I tought that at the end of the day the economic balance should have been positive.
  • I can see it now. Mesothelioma, YAZ and now McAfee lawsuit ads trolling for money.

1 1 was a race-horse, 2 2 was 1 2. When 1 1 1 1 race, 2 2 1 1 2.

Working...