Forgot your password?
typodupeerror
Security

ClamAV Forced Upgrade Breaks Email Servers 299

Posted by kdawson
from the on-the-half-shell dept.
An anonymous reader writes "A couple of weeks ago Sourcefire announced end-of-life for version 0.94 of its free ClamAV antivirus package (and in fact has been talking about it for six months). The method that Sourcefire chose to retire 0.94 was to shut down the server that provided its service. Those who had failed to upgrade are scrambling now. Many systems have no choice but to disable virus checking in order to continue to process email. I am very glad I saw the announcement last week!"
This discussion has been archived. No new comments can be posted.

ClamAV Forced Upgrade Breaks Email Servers

Comments Filter:
  • Alternative (Score:5, Insightful)

    by InsertWittyNameHere (1438813) on Friday April 16, 2010 @01:38PM (#31874160)
    The alternative was them not doing anything and then months later we see a story about how "ClamAV silently stops support. Virus outbreaks ensue."
    • Re:Alternative (Score:5, Insightful)

      by Anonymous Coward on Friday April 16, 2010 @01:42PM (#31874226)

      It's kind of an inflammatory article:

      Rather than simply phase this geriatric version out (it was at least one year old, revised to versions .95 and .96 since release, and announcements about the need to upgrade had been made for six months) the development team put to halt instances of V0.94 in production

      So, it's a year and two versions out of date AND they'd been saying for 6 months to move off it.. Yet still it's their fault for shutting down the server!? I'm sorry, but how much support do you want for something that's free?

      • Re:Alternative (Score:5, Informative)

        by compro01 (777531) on Friday April 16, 2010 @01:57PM (#31874452)

        It's quite a bit more extreme than just shutting down one of their servers. They issued a final "signature" update that literally caused each installation of that version to stop functioning.

        From the announcement [clamav.net] :

        Starting from 15 April 2010 our CVD will contain a special signature which disables all clamd installations older than 0.95 - that is to say older than 1 year.

        • Re:Alternative (Score:5, Insightful)

          by HarrySquatter (1698416) on Friday April 16, 2010 @02:04PM (#31874570)

          Would you trust an email server that is running a virus scanner that is more than a year out of date?

        • Re:Alternative (Score:5, Interesting)

          by ccandreva (409807) <chris@westnet.com> on Friday April 16, 2010 @02:13PM (#31874666) Homepage

          It's more complicated than that.

          Older versions of clamd were going to crash on signatures that newer versions would accept, and they have been prevented for at least 6 months from using that type of signature. They have posted since then for people to upgrade.

          When they did was publish this type of signature (has to do with length, greater than about 900bytes), where the signature itself is an error message, so when the program dumped the signature the error would be displayed.

          That's all, not a kill switch as such, but using a known bug to deliver a message, rather than have it just bomb out with a hex dump when they tried to use a larger signature.

        • Re: (Score:2, Interesting)

          by Anonymous Coward

          Yep, and when did they post that? 6 months ago. McAfee recently gave us 2 months notice at work that pre 8.x client would no longer be supported - not a problem as 7.1 was eol ages ago - since then there's been 8.0, 8.5 and currently 8.7 which we're moving to.

          No big deal for those who properly manage their systems.

  • by gparent (1242548) on Friday April 16, 2010 @01:39PM (#31874184)
    And you didn't, and now are going to complain when shit doesn't work? Go fuck yourself.
    • Re: (Score:3, Funny)

      by Anonymous Coward

      go fuck yourself

      uh. this is slashdot. for most of us, that is a redundant instruction.

      what would have been far more offensive is

      go fuck someone else

      as we all know that's not possible for most of us. ...you insensitive clod.

    • by johnshirley (709044) on Friday April 16, 2010 @02:04PM (#31874568) Homepage

      Kinda my attitude, too. Had this affect a bunch of servers yesterday. Started researching, found the cause, and solved the problem in 30 minutes on 35 or so servers. Totally my own damned fault for not staying upgraded. Worst impact was that messages were delayed on a few mail server for half an hour and uploads to a handful of webservers threw errors because of the way I scan them. Users tried again. Problem solved.

    • Re: (Score:2, Insightful)

      by The Moof (859402)

      So you had 6 months to upgrade and you didn't, and now are going to complain when shit doesn't work?

      No, but they'll complain (rightfully so) when the developers issue a "killswitch" command causing the software to quit working. So it's not like the servers disappear and stuff broke from obsolescence, they issued a command to the servers and had the software shut itself down (documented here [clamav.net]).

  • by WrongSizeGlass (838941) on Friday April 16, 2010 @01:40PM (#31874206)

    Diagnostic-Code: smtp;
    451-4.5.0 Error in processing, id=02792-02, virus_scan FAILED: virus_scan: ALL VIRUS SCANNERS FAILED: ClamAV-clamd av-scanner FAILED: CODE(0x83d7540) Too many retries to talk to /var/spool/amavisd/clamd.sock (Can't connect to UNIX socket /var/spool/amavisd/clamd.sock: No such file or directory) at (eval 55) line 310.

    ClamAV-clamscan av-scanner FAILED: /usr/bin/clamscan unexpected exit 50, output="LibClamAV Error: cli_hex2str(): Malformed hexstring: This ClamAV version has reached End of Life! Please upgrade to version 0.95 or later.

    At least their error messages are descriptive and informative.

    • Re: (Score:2, Flamebait)

      by thsths (31372)

      > At least their error messages are descriptive and informative.

      Indeed. Accurate error messages are something that Microsoft never quite achieved, and Apple never even tried. "It does not work, please have a look at our website www.fuckandall.com for possible causes" - I hate that!

  • [clamav-announce] (Score:5, Insightful)

    by 0racle (667029) on Friday April 16, 2010 @01:40PM (#31874208)
    It exists for a reason.
    • by 1s44c (552956)

      It exists for a reason.

      I'm going to subscribe to it now. I don't want to go though that again.

      But I can't subscribe to the announce list for every free software product I use, I'd do nothing else but read these lists.

      • Re:[clamav-announce] (Score:5, Informative)

        by entrigant (233266) on Friday April 16, 2010 @01:52PM (#31874378)

        announce lists are intentionally very low traffic. I'm subscribed to over 50, and I rarely receive more than 4 or 5 mails a week at most.

      • by 0racle (667029)
        This is what e-mail rules are for and to echo what the other poster said, they do not generate much traffic. While there is probably very little reason to subscribe to lists for absolutely every piece of software you run, you should probably subscribe to the announce lists for the major products you use.
  • this is common (Score:5, Insightful)

    by digitalsushi (137809) <slashdot@digitalsushi.com> on Friday April 16, 2010 @01:42PM (#31874236) Journal

    This is what we get when we're all our own "netadmins". I'm one of them. I don't follow security lists. I don't upgrade my products. Why not? Because I'm not really a netadmin. I just have a little server that runs until it breaks. I think that's the difference between a netadmin and a fake netadmin -- a fake netadmin like me reacts. A real netadmin is proactive.

    Which honestly, as pathetic as it sounds on the surface, works fairly well when your data and uptime don't matter. Because it's not pathetic because I have better things to do with my time than "run the family webserver".

    • by xaxa (988988)

      I got bored with being a "netadmin" once I started university. I moved my family's email to Google Apps, stopped giving free webspace to anyone that didn't already know what "SSH" meant, and haven't regretted it one bit.

      I do still have the server, but it only runs Apache. I looked into hosting, but I use ~20GB for photographs. Hosting for that is too expensive.

      (Although, I did run aptitude dist-upgrade every couple of months so probably wouldn't have been hit by this problem.)

    • This is why you rely on package management software. There are actual maintainers out there who keep up-to-date on issues like this, that affect their packages.

      For instance, if you're running any version of Ubuntu, you are on v0.95.3 or v0.96 [ubuntu.com] right now, so you would not have even known about this EOL had it not been on slashdot. Every time you log into Ubuntu, it will warn you if you need to do some updates.

      If you are not a professional system administrator (neither am I, by the way, so I feel for you), y

  • No fallback ? (Score:5, Insightful)

    by morcego (260031) on Friday April 16, 2010 @01:43PM (#31874238)

    People with critical servers that don't have fallback configurations to handle this kind of thing deserve to have their servers shutdown.

    I've been using 0.95 for some time now, so none of my servers were affected but, even if they were, my servers are smart enough not to interrupt the services, and to notify me.

    It is really disgusting the way people build servers these days. They think all they need to do is to install a couple packages, change a couple config lines and boom, the server is ready. They are getting what they asked for when stuff like this happens.

    • by 0racle (667029)
      I don't know, I think I'd rather mail pile up in the queue if my spam or AV product broke. I think I'd do something like this on purpose.
      • by Fiznarp (233)

        Yeah, noone really got hurt here.. just some delayed mail. I logged into my effected server and had clamav upgraded in 10 mins. It wasn't ideal but now I know I should have subscribed to the mailing list!

      • by morcego (260031)

        "Passing e-mails without checking in case the AV failed" is not really a fallback, at least not one I would recommend.

        I was talking about having a second, different AV for that.

    • Re: (Score:3, Informative)

      by 1s44c (552956)

      I had two mail servers, on two Internet connections. If either went down I'd get an alert and could fix it without mail being affected. I didn't expect both to stop processing mail at the same time. It's always the stuff you don't expect to fail that fails.

      My mail was queued on DMZ mailers so nothing was lost, but it was delayed. Some of it may have been business critical.

  • *Correction* (Score:5, Interesting)

    by Slipped_Disk (532132) on Friday April 16, 2010 @01:43PM (#31874242) Homepage Journal

    The method SourceFire chose to use was to encode a kill command in the ClamAV updates. If they had simply "shut down the [update] server" ClamAV would have continued to work, just without new signatures.

    See their announcement at http://www.clamav.net/lang/en/2009/10/05/eol-clamav-094/ [clamav.net]

    • Re: (Score:3, Informative)

      From the link:

      Starting from 15 April 2010 our CVD will contain a special signature which disables all clamd installations older than 0.95 – that is to say older than 1 year.

      [snip]

      We recommend that you always run the latest version of ClamAV to get optimal protection, reliability and performance.

      Thanks for your cooperation!

      FYI, ClamAV, DOA != cooperation.

    • So, they did the right thing. What is the big deal?

    • Re:*Correction* (Score:5, Insightful)

      by Anonymous Coward on Friday April 16, 2010 @01:59PM (#31874500)

      Wow. They could have just stopped publishing updates for older versions; they do have some method of versioning, right?. Older installations could have kept chugging along using the older definitions and newer installations could get the newer definitions. But to remotely *DISABLE* older installations? I don't care if the product and service is free or not; that is pretty fucked up.

    • That’s a very good thing to point outstill, though, it’s certainly not fair that having ClamAV get administratively killed from afar means that your email service coughs and dies.

    • Thank you.

      I would have been happy as a ... clam... if the way this went down was for me to simply find my log files full of warnings this morning.

      Instead, SourceFire chose to willfully break people's mail configurations, causing a huge amount of stress for those of us who are mail system maintainers.

  • by fuzzyfuzzyfungus (1223518) on Friday April 16, 2010 @01:44PM (#31874272) Journal
    Should have switched to Norton. They would have had weeks of impossible-to-ignore yellow and black pop-ups demanding their credit card number as ample warning...

    Those freetards just don't understand the valuable features provided by quality proprietary software.
  • by bogaboga (793279)

    ...and guess what! I'm almost sure I have had enough of free software.

    Not to say that it odes not do its work but because there is no incentive "not to break stuff", read 'continued revenue streams', folks just do as they please and we get hurt.

    Heck! Is this the "freedom" you want?

    • Heck! Is this the "freedom" you want?

      What, the freedom for your system to be very slightly unstable if you fail to upgrade a piece of software a year out of date after six months of warnings?

    • by NuShrike (561140)

      Yes, because you're not paying for it! Do you expect companies who have to make a buck be nice to leechers like you?

      When was the last time you donated to OpenBSD for all their contributions such as OpenSSH? If so many of your are going to be evil leechers, then companies have no choice and all the say.

    • by morcego (260031)

      You know the "free" part there doesn't mean you are free not to do a good job, right ? Because, you know, you are not.

      People still should know what they are doing. I never saw this announcement regarding 0.94, but nevertheless, none of my servers stopped.

    • Heck! Is this the "freedom" you want?

      Yes, thanks. While I have seen some frustrating breakages in OSS before (I recall several different Ubuntu updates that broke Xorg, the bastards), this isn't one of them. The software is a year out of date. You're given six months warning. Continuing to run after that time (if it were possible) would mean that your long-outdated version is no longer receiving definition updates -- so you'd be left with a false sense of security that you're somehow protected when you weren't.

      if they had just issued a r

    • by Lumpy (12016)

      Why because you were too lazy to update your AV software from a year ago?

      ClamAV did the right thing, they could have simply shoved out the new AV database that would have had your AV crash with a wierd error, because your horribly out of date version was incompatible with the new larger database format. but no they made sure you had a informative error so you would know what to do.

      But it's their fault and OSS fault... DAMN THOSE OSS PEOPLE!

    • by pizzach (1011925)
      Um. Then get something like RHEL and be done with it [mail-archive.com]. They specialize in keeping everything as stable as possible (and yes you pay for it.) It sounds like you are using the wrong product for your needs.
    • by Kijori (897770)

      ...and guess what! I'm almost sure I have had enough of free software.

      Not to say that it odes not do its work but because there is no incentive "not to break stuff", read 'continued revenue streams', folks just do as they please and we get hurt.

      Heck! Is this the "freedom" you want?

      For six months their web site, the clamav-announce mailing list and your log files have over and over again explained that the version was out of date and would be discontinued; it's not like this just happened overnight. But that's not even the point.

      The point is that this was in your best interests, although it may not seem like it now. Given that you hadn't updated for six months they could be pretty sure you weren't going to upgrade now; most likely you don't check the log files or the mailing lists bec

  • If it breaks because a remote server went away it sounds like it is time to possibly have another look at that code.

    • Re: (Score:2, Informative)

      by mysidia (191772)

      It wasn't the server going away. They delivered an update designed to kill it

      The Windows equivalent would be Microsoft Delivering a critical update with XP designed to disable windows, because you haven't updated to Vista yet.

      In other words, they used the automatic update service against their own users.

      From now on, my recommended course of action is that all mail administrators running clamav should REMOVE or DISABLE any automatic updates of ClamAV rules, make sure to comment out any crontab entri

      • by Nasarius (593729)

        The Windows equivalent would be Microsoft Delivering a critical update with XP designed to disable windows, because you haven't updated to Vista yet.

        No, not even remotely close. Upgrading ClamAV is trivial and costs nothing. If you're not keeping your security software up to date, you've failed utterly.

      • by Lumpy (12016) on Friday April 16, 2010 @02:27PM (#31874892) Homepage

        Nice FUD. the new DB will break it anyways.. and YES microsoft does this.

        They crafted a DB update that used that bug to deliver a message so the logs showed you what happened instead of a "seg fault - error in line 45867"

      • What were they supposed to do exactly?

        They've been warning users for 6 months that this was coming. The new style signature files for .95 and up were GOING to crash .94 installations. They're mirrors can't support supplying both old and new style signatures and the .95+ clients would have been _less secure_ because of a constrained signature file size. On top of all that if you'd go read their statement they ALSO cannot support an auto upgrade to .95 because of server constraints.

        Also, I have a feeling that

      • From now on, my recommended course of action is that all mail administrators running clamav should REMOVE or DISABLE any automatic updates of ClamAV rules, make sure to comment out any crontab entries for freshclam.

        <SARCASM>
        Mmhmm, yes. I agree 1000%. Don't update your virus signatures. Because ya know, new viruses don't get created very often. You can run with signatures over a year old and still have great protection!
        </SARCASM>

        Or do what they should do... include a method for automatically applying version updates.

        Or force auto version update instead of disabling.

        <SARCASM>
        Yes, because distributing software for several versions of Free/Net/OpenBSD, each Linux distribution, Windows, Solaris, AIX, HP-UX, etc. is totally feasible for a free project.

        It's not like they would have to fund the time, equipment and distribution bandwidth for t

      • From now on, my recommended course of action is that all mail administrators running clamav should REMOVE or DISABLE any automatic updates of ClamAV rules, make sure to comment out any crontab entries for freshclam.

        If you're not going to keep the virus signatures up to date, what's the point of even running it?

        It's a little shocking to me that anyone was caught by surprise on this. Ubuntu and Debian volatile are running 0.95+. I assume the other distros are, as well.

        If you don't intend to apply the security fixes to your server, do not run a server. Pay somebody else to do it for you.

    • by compro01 (777531)

      It isn't a remote server shutting down, they issued a "signature" update that caused each installation of a version prior to 0.95 to stop functioning.

    • by X0563511 (793323)

      You could try taking another look at the problem.

      The server is up. It specifically tells 0.94.x and earlier that "thou art broken"

  • by Anonymous Coward on Friday April 16, 2010 @01:50PM (#31874346)

    End of Life Announcement: ClamAV 0.94.x
    Oct 5, 2009

    All ClamAV releases older than 0.95 are affected by a bug in freshclam which prevents incremental updates from working with signatures longer than 980 bytes.
    You can find more details on this issue on our bugzilla (see bug #1395)

    This move is needed to push more people to upgrade to 0.95 .
    We would like to keep on supporting all old versions of our engine, but unfortunately this is no longer possible without causing a disservice to people running a recent release of ClamAV.
    The traffic generated by a full CVD download, as opposed to an incremental update, cannot be sustained by our mirrors.

    We plan to start releasing signatures which exceed the 980 bytes limit on May 2010.

    We recommend that you always run the latest version of ClamAV to get optimal protection, reliability and performance.

    Thanks for your cooperation!

  • by Knara (9377)
    IIRC, ClamAV doesn't have real-time scanning anyway. Does it have a first party mail server scanning plugin now, or am I totally misunderstanding the issue here.
  • Debian Debs Outdated (Score:5, Informative)

    by TypoNAM (695420) on Friday April 16, 2010 @02:05PM (#31874582)
    I just tried to update:

    # cat /etc/debian_version
    5.0.4

    aptitude output during update:

    Setting up clamav-daemon (0.94.dfsg.2-1lenny2) ...
    Starting ClamAV daemon: clamd LibClamAV Warning:
    LibClamAV Warning: *** This version of the ClamAV engine is outdated. ***
    LibClamAV Warning: *** DON'T PANIC! Read http://www.clamav.net/support/faq ***
    LibClamAV Warning:
    LibClamAV Error: cli_hex2str(): Malformed hexstring: This ClamAV version has reached End of Life! Please upgrade to version 0.95 or later. For more information see www.clamav.net/eol-clamav-094 and www.clamav.net/download (length: 169)
    LibClamAV Error: Problem parsing database at line 742
    LibClamAV Error: Can't load daily.ndb: Malformed database
    LibClamAV Error: cli_tgzload: Can't load daily.ndb
    LibClamAV Error: Can't load /var/lib/clamav/daily.cld: Malformed database
    ERROR: Malformed database

    It appears debian repositories also need to be updated. :(

    NOTE: I removed the * (star) chars from the warnings due to junk filter.
    • Re: (Score:3, Informative)

      by iYk6 (1425255)

      The ClamAV package in Debian Lenny-Volatile is 0.95.3. You're using the package from Debian Lenny, which is stable, and doesn't mesh well with ClamAV, which is either the latest and greatest or broken.

      Debian Volatile is meant specifically for this kind of thing.

      • Debian Volatile is meant specifically for this kind of thing.

        And indeed I'm running stable-volatile for my mail server, so I never would have found out about this, had it not been posted to slashdot.

        But it is truly shocking to me that Debian lenny hasn't been updated via security.debian.org. I know they're under a freeze and all, but there are about a half dozen bugs filed against clamav that warned this was going to happen. Not sure what the logic was in refusing to upgrade, despite this being a well-known to the maintainer issue.

        If they don't want to keep clamav

    • It appears debian repositories also need to be updated. :(

      In general, you may safely assume that to be the case for any given package.

    • It appears debian repositories also need to be updated. :(

      Follow the instructions here [debian.org] and then do the update. You'll be up and running in a jiffy.

  • by wolrahnaes (632574) <.sean. .at. .seanharlow.info.> on Friday April 16, 2010 @02:12PM (#31874658) Homepage Journal

    First you complain when Microsoft releases an update that won't install on compromised systems because it would break them entirely.

    Now ClamAV is put in a similar position. They have three choices due to the bug in 0.94:
    1. Continue supporting 0.94, flood out their update servers with full updates since incrementals won't work with that version much longer.
    2. Stop supporting 0.94, leaving users who don't know to update basically unprotected.
    3. Send a clear message to users who haven't updated that their antivirus solution is now broken and they need to upgrade.

    To me, 3 is the obvious choice. If this was a paid solution or if it cost a fucking dime to upgrade I might see a point to complaining, but to anyone who was still using 0.94 just man the fuck up, apt-get update, apt-get upgrade, and get on with it.

    This is not like Microsoft disabling XP to get you to upgrade to Vista, this is more comparable to an aircraft with faulty parts being grounded by the FAA. Those using 0.94 were doomed to a broken solution one way or another, they could not continue using it and expect it to do its job, so they needed a kick in the ass to upgrade.

    • wolrahnaes is exactly right. ClamAV was put in a position where they could easily end up with many email servers running with out of date antivirus definitions, but still think everything was working great. That is far more serious of a situation then stalling a few peoples email queues to force them to update. Had they silently stopped updating it would be way to easy for newly written viruses to spread because you would have such a large group of people who thought they were protected but weren't.
  • Overconfidence (Score:3, Informative)

    by gmuslera (3436) on Friday April 16, 2010 @02:15PM (#31874696) Homepage Journal
    A lot of server stuff in linux work so well that you can even forget that it is running at all, for years. Clamav is such kind of software, you install/configure it, set the automatic signature updates, and forget that it is there. But still, some periodic checks in logs that all are working as expected is good, even if is just some artificial ignorance [ranum.com] well applied, specially when clamav started warning on this months ago.
  • Misleading, yes? (Score:3, Informative)

    by thePowerOfGrayskull (905905) <(moc.liamg) (ta) (esidarap.cram)> on Friday April 16, 2010 @02:17PM (#31874744) Homepage Journal
    "ClamAV forced upgrade breaks email servers" should read "Failure to upgrade despite six months warning breaks email servers" or "Inattentive server admins cause massive downtime".

Genius is one percent inspiration and ninety-nine percent perspiration. -- Thomas Alva Edison

Working...