Please create an account to participate in the Slashdot moderation system

 



Forgot your password?
typodupeerror
Bug Businesses Red Hat Software Software Linux

RHN Bind Update Brings Down RHEL Named 312

Posted by kdawson
from the remind-me-of-your-name-again dept.
alexs writes "Red Hat's response to update bind through RHN, patching the DNS hole, made a fatal error which will revert all name servers to caching only servers. This meant that anyone running their own DNS service promptly lost all of their DNS records for which they were acting as primary or secondary name servers. Expect quite a few services provided by servers running RHEL to, errr, die until their system administrators can restore their named.conf. Instead of installing etc/named.conf to etc/named.rpmnew, Red Hat moved the current etc/named.conf to etc/named.conf.rpmsave and replaced etc/named.conf with the default caching only configuration. The fix is easy enough, but this is a schoolboy error which I am surprised Red Hat made. Unfortunately we were hit and our servers went down overnight while RHN dropped its bomb and I am frankly surprised there has not been more of an uproar about this."
This discussion has been archived. No new comments can be posted.

RHN Bind Update Brings Down RHEL Named

Comments Filter:
  • by Anonymous Coward on Friday July 18, 2008 @07:17AM (#24240283)

    So, you didn't test the update on a non-production server? Just install any old patch and let it take your network down? Who do you work for again? I have to make sure not to do business with that.

    • by suso (153703) * on Friday July 18, 2008 @08:03AM (#24240753) Homepage Journal

      Actually, I caught the error just from looking at the output of up2date/yum. It clearly said named.conf saved to named.conf.rpmsave. So all you have to do is compare what changed, implement any changes and copy named.conf.rpmsave over named.conf.

        Just as I said on the day of the release, be careful, don't just blindly update things.

    • by illumin8 (148082) on Friday July 18, 2008 @08:26AM (#24241067) Journal

      So, you didn't test the update on a non-production server? Just install any old patch and let it take your network down? Who do you work for again? I have to make sure not to do business with that.

      No kidding. The only "schoolboy error" as the submitter put it, was not testing the patch on a non-production server before deploying it on a production DNS server.

      • by xalorous (883991)

        People make mistakes. Even _________ (insert linux vendor here) packagers. This patch should be remembered and trotted out every time a new sysadmin is taught how to do the job. Remember the DNS bind patch of '08? That's why you test before patching production servers.

        Oh, and the 'I don't have enough money for a spare of every server' excuse won't play. That's one reason to buy consistent models. Your test equipment can serve as emergency replacements or vice versa. And if all else fails, testing on

      • Re: (Score:3, Interesting)

        by mi (197448)

        The only "schoolboy error" [...] was not testing the patch on a non-production server before deploying it on a production

        Can the same line be used to defend Microsoft the next time they screw up a bug-fix or "service pack"?

    • Re: (Score:3, Insightful)

      by jocknerd (29758)

      You know, not everyone has non-production servers. Every server we have IS production. And if you are paying for Red Hat Enterprise, you expect Red Hat to have tested these updates themselves. If this was a Microsoft error, Slashdot would be all over Microsoft for allowing this to happen.

      • by numbsafari (139135) <swilson.bsd4us@org> on Friday July 18, 2008 @09:15AM (#24241807)

        And the rest of slashdot would be all over MS admins who blindly update their systems from AutoUpdate.

        I find it really hard to believe you don't have at the very least a strawman test system. The fact that you don't says volumes.

      • by bucky0 (229117)

        To be sure, red hat ballsed it up, but if you're running a service that "can't go down", you HAVE to test your patches out. If you don't have spare physical machines, test them in a virtual machine or repartion your workstation to have enough room for a server install.

        If it's important enough to not go down, it's important enough to test.

        • Re: (Score:2, Insightful)

          by GleeBot (1301227)

          And contrariwise, if it's not important enough to test, then it's not important enough to not go down. So grin and bear it.

      • by Sleepy (4551) on Friday July 18, 2008 @09:27AM (#24242003) Homepage

        >You know, not everyone has non-production servers. Every server we have IS production. And if you are paying for Red Hat Enterprise, you expect Red Hat to have tested these updates themselves. If this was a Microsoft error, Slashdot would be all over Microsoft for allowing this to happen.

        You are wrong; stop whining. You're just painting yourself as misinformed.

        1) The updates WERE tested.
        2) The admin installed "caching-nameserver", then configured his install to act far outside the default.
        3) He allows automatic updates straight into production. So do you it seems. Good luck with that! RHEL documentation says to not do this, but you're a bigshot "paying" for something different. I suggest you get a sidekick, and stick to the Windows side of your "enterprise".
        4) He didn't revert his .conf file, as is usually needed when some new line is added to a server .conf. This is SO NORMAL you'd have to be a n00b to get bitten!

        Your MS comparison is apples and oranges. If this guy did TEN MINUTES worth of testing he'd realize something's up, and he could revert the rpm package. How many MS updates prohibit uninstall? Quite a few!

        In Windows, you can't diff the before & after config, since Windows admins would rather be blind to what they're installing, since that's the norm and it's accepted.

      • by poot_rootbeer (188613) on Friday July 18, 2008 @09:31AM (#24242063)

        You know, not everyone has non-production servers. Every server we have IS production.

        Well, there's your problem right there...

        • Perhaps it is his problem, but not his fault. Sounds like he's in the dreaded zone where IT is a necessary evil, not a department that can help leverage the business.

          He gets what he needs, or just barely what he needs. When management hands you crap, you learn to make crapade.

          • Re: (Score:3, Insightful)

            by ebuck (585470)

            You are right about making do with what you get, but exactly how did he lack resources in this case? He already has RHEL (and updates, so I'm guessing his support contract is up to date).

            It's not like they're charging more for a non-caching domain name services server. In fact, he took a perfectly good non-caching name server, and then installed pre-packaged configuration files to make it a caching-nameserver. Then he started hacking away at the config file. Small wonder that fixes to the caching-namese

      • Re: (Score:2, Insightful)

        by IchNiSan (526249)
        You mean to tell me you don't even have an old desktop machine sitting around with RHEL on it to "play" with? Come on, pull the other leg. Or maybe find a new line of work. Not being able to afford non production servers and test lab is one thing, but not taking the old computer you replaced on the secretaries desk and using that to do some basic testing for mission critical updates is ridiculous. Or hell, just dual boot your machine if it comes to that. You have to do SOME testing of SOME things.
      • So run it on another machine that isnt an authoritiative nameserver?

        And/or, if your company is so broke it can't afford a couple hundred bucks to put together a low-end box to run update tests on, then you are doomed anyway.

        "Free Software", even when serviced and supported by a corporation such as RedHat, is about knowing WTF you are doing and being responsible for your own stuff, as opposed to being a drooling button-pusher assuming everyone else will take care of you and suing them when they don't.

        And if

  • A schoolboy error? (Score:5, Insightful)

    by something_wicked_thi (918168) on Friday July 18, 2008 @07:17AM (#24240285)

    What? And isn't it an error of similar proportion to upgrade your primary DNS servers without first testing the new install?

    • by imipak (254310) on Friday July 18, 2008 @07:38AM (#24240499) Journal

      Note as well that the initial release included a default conf file which specified a fixed source port, which of course breaks the fix.

      [Updated 10th July 2008] We have updated the Enterprise Linux 5 packages in this advisory. The default and sample caching-nameserver configuration files have been updated so that they do not specify a fixed query-source port. Administrators wishing to take advantage of randomized UDP source ports should check their configuration file to ensure they have not specified fixed query-source ports.

      Personally I'm surprised there's not been more uproar about the requirement to move internal DNS servers (yes, that means your Windows Domain Controllers in most corporate environments) outside any NAT'ing devices (eg: firewalls), as many NATs also break the fix by rewriting outbound UDP DNS queries to use the same or incremental source ports, which also breaks the fixes. Anyone here moved their AD outside the firewall?

      • Mod parent up! (Score:4, Insightful)

        by Chrisq (894406) on Friday July 18, 2008 @08:06AM (#24240783)
        I am sure that many people do not realise that going through a NAT device usually means that predictable port numbers will be allocated.

        Of course until we get details of the hole and fix we cannot be 100% sure but it is very likely that exposing predictable port numbers (which the fix randomised) reintroduces the hole.

        If DNS software vendors had a year's notice then why didn't the NAT firewall vendors. They could have introduced a patch at the same time.
        • by afidel (530433)
          Apparently [checkpoint.com] they did, either that or Checkpoint's protection feature for the general class of DNS poisoning attacks just happened to protect against this one too. However, even if it did protect against it I doubt they could have release a same day press release stating it did if they hadn't been notified of the vulnerability ahead of time.
        • Re: (Score:3, Informative)

          Judging by the CERN details, it sounds like there are two things you need to do. You need to be able to predict the 16-bit random number, and the 16-bit random port. My reading (and this was very brief, so someone *please* correct me if I'm wrong here) is that the older DNS servers had two flaws: a flaw in the RNG for the 16-bit transaction number, and they used fixed or predictable ports.

          A NAT will reintroduce only the second problem because it gives you predictable ports, but obviously, relying solely on

          • Re: (Score:2, Informative)

            by Anonymous Coward

            Judging by the CERN details, it sounds like there are two things you need to do. You need to be able to predict the 16-bit random number, and the 16-bit random port. My reading (and this was very brief, so someone *please* correct me if I'm wrong here) is that the older DNS servers had two flaws: a flaw in the RNG for the 16-bit transaction number, and they used fixed or predictable ports.

            A NAT will reintroduce only the second problem because it gives you predictable ports, but obviously, relying solely on the unpredictability of a 16-bit transaction id is a little scary. Because of the birthday paradox, (assuming the attacker has perfect knowledge about which port you're choosing) an attacker would need to send only something on the order of 2^8 packets to poison the cache.

            No, the birthday problem doesn't apply when you are trying to match a specific person's birthday.

        • Re:Mod parent up! (Score:4, Informative)

          by Effugas (2378) * on Friday July 18, 2008 @01:02PM (#24245331) Homepage

          [This is Dan Kaminsky]

          The NAT vendors didn't get as much notice because we didn't realize so many of them were doing this.

          If we had, they'd have been brought in from the start.

          Now they're scrambling, to their credit. It's a bit of a facepalm for me.

      • Re: (Score:2, Insightful)

        by evilviper (135110)

        Personally I'm surprised there's not been more uproar about the requirement to move internal DNS servers (yes, that means your Windows Domain Controllers in most corporate environments) outside any NAT'ing devices (eg: firewalls),

        NAT is not a firewall.
        A firewall is not NAT.

        I wouldn't think that practically any major sites are running their public-facing DNS servers from behind a NAT (though I expect most are behind a firewall).

      • by db32 (862117)
        Personally I'm surprised that this is getting modded Informative. I suppose the NAT piece is informative, but I think "Anyone here moved their AD outside the firewall?" qualifies as either -1 Job or +5 Funny.
      • by igb (28052) on Friday July 18, 2008 @08:36AM (#24241185)

        Hand off DNS queries emerging from AD servers inside your firewall to caching-only servers in your DMZ. I have all my AD servers on RFC1918 IP numbers with no NAT, because they strike me as devices I'd prefer to keep as far away from the big bad Internet as possible.

        ian

  • MS (Score:4, Insightful)

    by FozE_Bear (1093167) on Friday July 18, 2008 @07:17AM (#24240297)

    If it was a Microsoft product, we'd all be carrying pitchforks and torches....

    • by hughesjr (734512)
      sure, if it was really a bug ... in this case, it is totally user error and the installing of a package called caching-nameserver.
    • the thing youre forgetting is that microsoft REGULARLY does that, and even with irrelevant minor updates. thats why people are too worked up because of microsoft. they will gonna let this red hat incident slip by, because red hat doesnt have a track record of messing it up.
    • by debatem1 (1087307)
      Yup. I've got all the torches, can you grab an extra pitchfork? I'll pay you back when we get to the castle.

      Seems like the general opinion is that no admin worthy of avoiding the boiling oil treatment wouldn't have applied the patch blindly to a production environment, but it still doesn't let RedHat off the hook.
  • bug details (Score:5, Informative)

    by tommis (1328303) on Friday July 18, 2008 @07:23AM (#24240353)

    Here's the bug details: https://bugzilla.redhat.com/show_bug.cgi?id=453340 [redhat.com]

    One of the bug comments says: "Latest caching-nameserver renamed my named.conf to named.conf.rpmsave in /var/named/chroot/etc" - so this should mean that you can still restore the lost conf file.

    • Re: (Score:3, Informative)

      by Chris Mattern (191822)

      In other words, as somebody else posted, he installed the "caching-nameserver" package and got, surprise, a caching nameserver. Shocking.

    • Re:bug details (Score:5, Informative)

      by hughesjr (734512) on Friday July 18, 2008 @07:50AM (#24240641) Homepage
      it is not a bug to get a caching nameserver if you install caching-namesever ... it would be a bug to install caching-nameserver and NOT GET a caching nameserver.
      A caching name server IS one that does not have any zones and only looks up zones from the DNS root servers. It is a configuration error to install the caching-nameserver package on a machine that doing anything other being a caching name server.
      Stupid admins have been complaining about this for 5 years ... but the documentation and bug entries all make it clear NOT to install the caching-namesever packages on DNS servers that control zones.
      • I wish I had mod points with which to mod you up. This is NOT a bug, and a few RHEL test machines I have here updated just fine, keeping their zone files as expected.

      • Yes but a caching name server is (or at least was for a long time) the Red Hat default. Go figure. That's all most people want or need. Any bets that lots of the people who got bit with this built the machine internally with the caching name server RPM installed and then just edited or copied over the production named.conf file to turn it into their "real" name server when the box went into production?

        Cheers,
        Dave

  • Um... (Score:5, Funny)

    by wellingtonsteve (892855) <wellingtonsteve@@@gmail...com> on Friday July 18, 2008 @07:23AM (#24240355)

    "I am frankly surprised there has not been more of an uproar about this"

    That's because the entire Internets are now broken!

    • by exi1ed0ne (647852)

      That's because the entire Internets are now broken!

      It was as if millions of p0rn sites cried out in terror, and were suddenly silenced.

  • argh (Score:2, Insightful)

    by Fackamato (913248)
    I guess the syadmins could put in an option in a configuration file somewhere on what files to "keep untouched" when doing package upgrades, no? So that the configuration file wouldn't be overwritten. I think I've seen something similar in Debian distros. Anyway when I install a new (custom) kernel in Ubuntu for example, synaptic asks me if I want to overwrite GRUB's menu.lst with the newly generated one, view the differences or keep my old one etc. Surely there's something similar in Redhat?
  • by nicolas.kassis (875270) on Friday July 18, 2008 @07:24AM (#24240367)
    Half of whole point of a subscription to RHEL is to ensure that patches they put out are properly QAed. The other side is support, but I never had a chance to test that part out.
    • Re: (Score:3, Insightful)

      by MikeDawg (721537)
      Umm. . . I disagree completely. The only way I would consider a patch "put out properly" if it was tested in my exact, or near exact environment. I can only assume that I'm not important enough for that.
  • No worries (Score:2, Interesting)

    by FlyingBishop (1293238)

    I don't need to worry about that, I run Debian

    Also, I don't run my own DNS. But if I were paying someone to make sure my patches weren't idiotic, I'd be pretty pissed, whether the patch was for something I used or not.

  • You are WRONG :D (Score:5, Interesting)

    by hughesjr (734512) on Friday July 18, 2008 @07:26AM (#24240385) Homepage
    This article is absolutely wrong.

    The user has misconfigured their DNS and has installed a package called, SURPRISE, caching-nameserver along with the other bind packages.

    caching-nameserver IS just that, a caching-nameserver. It SHOULD NEVER BE installed on a DNS server that is used for Primary or Secondary DNS control. The bind packages do not in any way modify named.conf, but if you want a caching nameserver and if you have installed the caching-nameserver package, then you would EXPECT that it would replace the named.conf file.

    The real question is, how does crap like this get posted as a feature article on slashdot.
    • /. QA is worse than the poster's?
    • by Kamokazi (1080091)
      There is a reason a lot of articles used to get tagged 'kdawsonfud'
  • Test your patches (Score:3, Insightful)

    by MikeDawg (721537) on Friday July 18, 2008 @07:27AM (#24240395) Homepage Journal
    What kind of environment are you in where you don't first test your patches that are going out to live production machines? Regardless of the fact that it is linux and not windows, you should always test your patches before you roll them production.
    • by Just Some Guy (3352) <kirk+slashdot@strauser.com> on Friday July 18, 2008 @08:38AM (#24241217) Homepage Journal

      What kind of environment are you in where you don't first test your patches that are going out to live production machines? Regardless of the fact that it is linux and not windows, you should always test your patches before you roll them production.

      Disclaimer: I test first.

      You know, lot of people work in small shops that can't afford multiple redundant servers. I suspect that business with a single DNS/web/mailserver are a lot more common than Slashdotters this morning seem to thing. What are those admins supposed to do? They're receiving a critical security patch from a trusted vendor, and I imagine a lot of them feel pretty safe applying that to their sole production server. This doesn't make them stupid or incompetent.

      I have the luxury of lots of hardware that can fill in for other gear in a pinch, but lots of people don't. They don't deserve scorn for it.

      • by radish (98371)

        Couple of points. Firstly, the smart but under resourced admin should use this incident as evidence of what happens when management won't cough up for required equipment. A test environment is not a luxury, it's a necessity for a reliable system. Secondly, something like vmware can let you set up and test an environment on whatever hardware you do have lying around - you could even run it on the prod box at a pinch.

      • "lot of people work in small shops that can't afford multiple redundant servers .. What are those admins supposed to do?"

        Keep two harddrives in the same machine and keep a clone of the DNS server on the second HD, if an upgrade borks, just swap the drives.
      • by Sleepy (4551)

        You nailed the demographic, but these are EXACTLY the group that should not be running their OWN exposed servers.
        This would be true for any server, but "double" so for DNS.

        DNS hosting is cheap, and the expenses are unpredictable (unlike the commotion raised when you have "Windows IT" people whining about "what they pay Redhat for". UNIX does what you ask. If you want 10 meters of rope, tie it to a beam and stick your head in... it will let you.

        Automatic updates are FINE for the desktop, in shops like you de

  • A few months prior to the release of RHEL 5.2, they released a kernel update (2.6.18-53.1.6.el5) in which they had added a patch for an issue that could make a system oops upon when files with names of a certain character were present on NFS shares. However, this patch also contained a bug which broke NFS lookup caching and subsequently crippled NFS performance to the point of NFS being completely unusable when working with multiple smaller files. They released a patch for it, but it would only apply cleanl
  • ...check for rpm mouse droppings by running find.

    RH may have made a small coding mistake - you made an even bigger one.

  • by Spazmania (174582) on Friday July 18, 2008 @07:33AM (#24240445) Homepage

    Red Hat makes this mistake a LOT. It makes the update process very unreliable. SuSE isn't as bad but they still have problems if you customize a piece of software's configuration in an unexpected way.

    Debian is king here. The incremental patches almost never break a configuration and the major release upgrades tend to work; they often change package names if the new "version" has a major incompatible change in the configuration.

    • by houghi (78078)

      openSUSE has warnings in those files NOT to use them and tell you which one you need to use. e.g. for Apaches http.conf it says:
      # If possible, avoid changes to this file.
      It then goes on to exlain a lot about what to place where.

      There are many other files like that. Another example:
      # PLEASE DO NOT CHANGE /etc/bash.bashrc There are chances that your changes
      # will be lost during system upgrades. Instead use /etc/bash.bashrc.local
      # for your local settings, favourite global aliases, VISUAL and EDITOR
      # variables

      • by Spazmania (174582)

        openSUSE has warnings in those files NOT to use them and tell you which one you need to use.

        True enough. Doesn't help when I want the application to do something it is capable of but which wasn't envisioned by the SuSE packagers. Like binding sendmail to a non-privileged port as a non-privileged user and then using iptables to redirect port 25 up to that port.

        Debian won't automatically overwrite my modified init.d script during an upgrade. It'll ask permission with the default set to "no." SuSE and Red Hat

  • Well (Score:4, Insightful)

    by ledow (319597) on Friday July 18, 2008 @07:44AM (#24240569) Homepage

    Yeah, it's a silly mistake.

    But you should be testing things like this first, and whenever you upgrade you should really be looking at/for all .rpmsave or equivalent files first to make sure nothing has changed in the meantime. Otherwise, you're just removing your config and replacing it with the default whatever happens. You should also be checking .rpmnew (or equivalent) each time to check that it hasn't changed in terms of syntax, defaults etc. (which, let's be honest, is quite likely for such an important update - especially given that we hardly know what the exact problem is yet). I wouldn't go so far as to suggest intimate analysis of packages while they are still packed unless the systems you are running are quite critical to the operation of a business.

    Part human-error on RH's part (it happens). Part incompetence in not testing the updates yourself first. Chances are that if I were affected by this, I would catch it as part of "right, what did that package change?", or notice as part of usual testing later, and then just move the file. I probably wouldn't even bother to send RH a note.

    If you have a DNS server, that suggests that there are reliant computers. As courtesy to all those reliant computers you HAVE to test changes and check carefully what they are doing first. If you were "stung" by it, it suggests you hit this problem on ALL your DNS servers and/or that you only have one DNS server anyway. To deploy packages like this on such a setup is just asking for trouble.

    • Re: (Score:2, Informative)

      by LeneJ (190881)

      I just want to clarify a bit about rpmnew vs rpmsave.

      Red Hat will create an rpmsave file when we make a significant change to the configuration file, or a mandatory change. Other than that, we keep the original config file, and store the rpm-config as rpmnew.

  • Don't forget! (Score:5, Informative)

    by prandal (87280) on Friday July 18, 2008 @07:45AM (#24240579)

    Don't forget to check your named.conf on RHEL 5.x (and CentOS 5.x).

    Make sure that any lines like

    query-source port 53;
    query-source-v6 port 53;

    are commented out or deleted so that forwarded DNS queries come from random ports.

    Restart BIND if necessary.

  • by cluening (6626) on Friday July 18, 2008 @07:58AM (#24240703) Homepage

    Have you considered using a configuration management tool such as Bcfg2 [bcfg2.org] or cfengine to make sure your own config files are restored after package updates are made? You can never really trust those package maintainers...

    • Or even version control. As much love as Git and Subversion get around here, I'm surprised you don't hear more people advocating using them for config files.

  • Suprised... (Score:2, Redundant)

    by kenh (9056)

    I must say that I am very suprised that this patch acted one way in the posters test environment and another when it was installed on their production machine... That's very odd.

    What, he didn't test it before placing it in production? Never mind, move along - nothing to see here.

    If the poster made an error (as suggested by a previous post), or if he installed a patch without testing it, bad on the original poster - but if the patch truely was bad (a possibility), then bad on RHN for letting something bad ou

  • ...the file was backed up and not deleted.

  • "Unfortunately we were hit and our servers went down overnight while RHN dropped its bomb and I am frankly surprised there has not been more of an uproar about this"

    You mean you installed a patch that you didn't know was faulty and didn't have a rollback option in place, shame on *you* Mr. Sysadmin .. :)
  • Does the red hat version of apt-get (yum? I've used debian exclusively for a long time, so I forget what command it was) not prompt you when it wants to overwrite a config file? On any debian (or debian derived) machine I've used, apt-get always asks what you want to do if your config file is different than the package's.

    If yum(?) blows away configs without prompting, that's pretty bad.

  • by Venotar (233363) on Friday July 18, 2008 @09:50AM (#24242367) Homepage

    This is news? Redhat (like every OS vendor I've ever dealt with) have been pushing out updates with broken assumptions for years.

    In fact, this isn't even the first time they've done something similar when updating bind:
    back in 2004 they released RHEL 3 update 4 and many people had precisely the same experience. Additionally, when applied, Update 4 removed the /etc/rc*.d/S*named and /etc/rc*.d/K*named and then shut named off.

    As a quick glance at redhat's bugzilla [redhat.com] shows, the first problem (the same one you experienced in this release) wasn't a schoolboy mistake on the packagers part, or a bug. It was the result of a poorly understood choice on the part of the person who originally provisioned the machine.

    Rather than installing just the original bind-9.2.4, the people who had their named.conf overwritten had installed bind plus a package called caching-nameserver. It's that package that, when updated, backed up and overwrote their bind config. The "caching-nameserver" package should only be installed if you want to run a caching nameserver, because the caching-nameserver package isn't an application at all - it's simply a named.conf file.

    The real bug (back in 2004) wasn't actually in Update 4's bind package. As it turns out, the package it replaced incorrectly contained a `chkconfig --del named` in its uninstall script.

    Anyone without proper alerting and a good QA process found that one out the hard way. I had customers who'd gotten so blasè about performing nighttime maintenances without proper reversion testing that they scheduled nightly cronjobs that ran up2date at midnight and rebooted the production machine, Naturally, they woke up in the morning to find they'd just suffered 8 hours of downtime.

    Lesson? Don't trust the vendor's QC work, don't install unnecessary packages, and make sure to QC your own work! Ask any experienced Windows admin about unintended consequences from "trusted" vendor patches...

  • by not_hylas( ) (703994) on Friday July 18, 2008 @09:55AM (#24242447) Homepage Journal

    GUILTY.
    Seems the person that prepared the patch is a new hire at Red Hat.

    Beware Latest 10.3.x security update - it replaces /etc/named.conf:

    http://discussions.apple.com/message.jspa?messageID=5876624 [apple.com]

  • by Blimey85 (609949) on Friday July 18, 2008 @09:55AM (#24242455)
    Crap like this is why I lost faith in Microsoft and quit running Windows years ago. Thankfully my RHEL box isn't affected by this sort of... oh... wait... really? Shit.
  • Check here:

    http://lateral.netmanagers.com.ar/weblog/2008/07/16.html#BB701 [netmanagers.com.ar]

    In at least one very common config, named.conf is a symlink, so copying it doesn'tavoid it being overwritten.

    The named update script "copies" symlinks by making another symlink, not by copying the underlying file.

  • by Todd Knarr (15451) on Friday July 18, 2008 @10:17AM (#24242855) Homepage

    This sounds like how RPM's behaved as long as I can remember. It looks at three versions of a config file: #1 the one from the old package, #2 the one currently on disk and #3 the one in the new package. If the config file hasn't been customized (1 and 2 are identical), it moves the old file to .rpmold (if 1 and 3 differ) and puts #3 into place. If the config file has been customized, it checks whether 1 and 3 differ. If they haven't then nothing's chanced, the customized config file's still valid and it drops #3 in with the .rpmnew extension. But if 1 and 3 differ, then something in the config file may have changed and the customized config file may no longer be valid. But it's got customizations in it that the admin may need to refer to. So it outputs a warning message about what it's doing, moves the customized config file to .rpmsave and installs #3, and the admin's expected to have seen the warning and to merge their customizations into the new config file. You do watch for warnings and errors during the update, right?

    In this case RPM is right, old named.conf files aren't valid. If they're based off RH's old stock config files, they have the source port locked and that disables much of the security fix. So the admins do have to check and modify their customized files before the system's finally ready (or at least RPM has to assume they do, since it can't know exactly what their changes were). That's exacerbated by probably having caching-nameserver installed, but I think a stock BIND install has a similar named.conf until you add your own zones to it.

    I'd chalk this one up to admins who a) don't understand an inherent limitation of package-management systems (namely, it doesn't know why you changed something, only that you changed it), b) didn't watch the update process for errors, and c) didn't check the systems for functionality after the update.

  • ...that neither 'bind' nor 'named' should be capitalized. Then again, they're not very technical people.
  • Oh so LATE (Score:3, Interesting)

    by alexborges (313924) on Friday July 18, 2008 @11:03AM (#24243613)

    Thanks ./, ive known about this for TWO WEEKS.

    And no one died.

    So there.

  • by mi (197448) <slashdot-2014@virtual-estates.net> on Friday July 18, 2008 @11:15AM (#24243793) Homepage

    Don't entrust the function like DNS to a single vendor. With some services it is hard, as authors support a limited range of OSes/hardware or charge too high a price for each installation to make redundancy affordable.

    But not DNS. Free solutions abound, and the commercial ones are quite cheap too. They are available for all imaginable "server-grade" OS/hardware combination. If you use more than one servers for DNS in your enterprise, and both of them use the same platform, you aren't doing your job.

    Mind you, I don't blame the victims here — Red Hat screwed up royally, and that's that. Just advising on how to avoid being hit by such (inevitable) mistakes — from any vendor — in the future.

Building translators is good clean fun. -- T. Cheatham

Working...