Forgot your password?
typodupeerror
Bug HP Networking

HP Server Killer Firmware Update On the Loose 100

Posted by timothy
from the just-this-one-little-problem dept.
OffTheLip (636691) writes "According to a Customer Advisory released by HP and reported on by the Channel Register website, a recently released firmware update for the ubiquitous HP Proliant server line could disable the network capability of affected systems. Broadcom NICs in G2-G7 servers are identified as potentially vulnerable. The release date for the firmware was April 18 so expect the number of systems affected to go up. HP has not released the number of systems vulnerable to the update."
This discussion has been archived. No new comments can be posted.

HP Server Killer Firmware Update On the Loose

Comments Filter:
  • Hot firmware (Score:5, Interesting)

    by DigiShaman (671371) on Friday April 25, 2014 @08:59AM (#46840053) Homepage

    And this is why I wait at least 2 months before installing firmware updates (unless it's a major security issue). It's not uncommon for a firmware update to be pulled shortly after being published. The 2 month window delay is generally more then enough time to ensure it's a proper update is solid.

    • by SJHillman (1966756) on Friday April 25, 2014 @09:57AM (#46840467)

      The company budget committee helps us avoid issues like this. The majority of our gear is old enough that there hasn't been a firmware update in four or five years. And no plans to replace it any time soon.

    • by Culture20 (968837)
      But this firmware is killer, which in software terms is the same as bodacious!
      P.S. I find it sad that bodacious isn't in my browser's or OS' dictionary files.
    • by Lumpy (12016)

      I wait until I HAVE TO install a firmware update. Unless there is a major problem that will cause the server to explode, you dont update the firmware.

      • I would say the most dicy of them all is HDD (generally SAS) firmware updates. A car analogy with else specific firmware updates is like changing the automatic transmission oil. If you update (change) HDD firmware as they become available, you prolong the life of what would otherwise be premature failing hardware. Otherwise if you never did any maintenance towards the end of their life, and then pushed an update through, you may find that the drive failed shortly after. Again, this *only* applies to HDD fir

      • by idontgno (624372)

        How 'bout it's a major problem because the company won't honor its warranty and support contract until you slavishly install every update?

        The good news, though, is that after this update you'll probably have a better reason to open a ticket than the piddly-ass one you had before.

    • by AmiMoJo (196126) *

      But didn't you hear? It's on the loose, it could be anywhere and upgrade your servers at any time!

    • I deployed a DL380p Gen8 last year, and it gave me heart failure.

      Under Red Hat, I needed to change the IP address, so I modified the file /etc/sysconfig/network-scripts/ifcfg-eth0 then did a "service network restart"

      Alas, the box did not come up on the new IP. Got to the console which was blank and unresponsive. Power cycled, and the RAID array was GONE (and let's just say this was EXTREMELY inconvenient timing).

      Support was able to walk us through some BIOS disk recovery that (thankfully) worked. But I'll n

  • We pushed a firmware update this morning to the firewall and its been smooth sai#*($$#[NO CARRIER]
    • by operagost (62405)
      Fantastic old-school post from someone with a high six-digit ID. Although, I guess a sixer counts as a old-timer now.
    • by idontgno (624372)

      [NO CARRIER]

      There's your problem. You flashed your firewall with the firmware package for a mid-90s Hayes-compatible telephone modem. I hope you have a spare; that firewall is hung up good.

  • by Rich0 (548339) on Friday April 25, 2014 @09:01AM (#46840057) Homepage

    ...don't flash it.

    Do admins routinely flash firmware updates in the absence of some identified need? I could see flashing an update if I was suffering from a known problem, or if the vendor identified a security flaw in a previous release. I could see flashing it if necessary to install new hardware.

    I just don't see why a server admin would flash a firmware update as if it were Patch Tuesday. In the absence of a security vulnerability or production issue there is no reason to treat a firmware change as an expedited change and not perform full testing before deploying it. That isn't to say that doing some testing of security patches/etc isn't wise - but I can see why it would get rushed.

    • and when your entire site goes down on a Monday morning because one of your vendors applied an update to some connecting hardware? And their response when asked for the reason for the outage is "Your hardware was 3yrs out of date. Your Sys Admin said it wasn't broke so he didn't fix it" What's your boss going to say after he gets done telling how many years of your salary the outage cost the company?

      I delay updates, but I get that shit approved by executive officers first. I always make sure I have a very g

      • and when your entire site goes down on a Monday morning because one of your vendors applied an update to some connecting hardware? And their response when asked for the reason for the outage is "Your hardware was 3yrs out of date. Your Sys Admin said it wasn't broke so he didn't fix it" What's your boss going to say after he gets done telling how many years of your salary the outage cost the company?

        I delay updates, but I get that shit approved by executive officers first. I always make sure I have a very good reason to delay it as well.

        Ah yes, Sys Admin, the damned if you do and damned if you don't profession.

        • This is why I believe in half-assing everything. If it's only half done, you can only be half damned either way. Right?

      • by Lumpy (12016)

        My boss will be screaming at the vendor, they are not allowed to push any updates until we approve them.

        • My boss will be screaming at the vendor, they are not allowed to push any updates until we approve them.

          Allowed? hahahaha...
          Your vendor has a contractual obligation to... well... follow the contract. The contract does what the contract says... if they screw up, you get money off the bill. It doesn't stop their lead tech from going on a drunken binge after his wife leaves him and taking a golf club to the equipment.

          • by Lumpy (12016)

            What moron would sign a contract with a vendor like that, you have drooling morons working as management for your company?

    • by MrNemesis (587188) on Friday April 25, 2014 @09:22AM (#46840209) Homepage Journal

      TBH, I suspect this is just getting publicity since it's the first super-dodgy HP firmware patch since they adopted their "no updates for YOU!" mentality - the explanation for which from HP was that they'd sunk a lot of money into their patching process and people shouldn't get to use it for free I guess. This won't be the last time this happens either.

      As a sysadmin that's dealt with dozens of these "killer firmwares", there's often an indentified need. We make extensive use of the HP SPP's at work and they come with a list of fixes and known issues as long as your arm; it's part of my job to go through the advisories to see if we're at risk and if we are to analyse the risk of updating/not updating. Many of them aren't security vulns or emergency fixes and are often extremely obscure, but once in a while you'll encounter something like a NIC locking up on receiving a certain type of packet or the BIOS repeatedly saying a DIMM has failed when it hasn't, or if you mix hard drives with firmware X and firmware Y on RAID controller Z running firmware... er.. A it might drop the whole array... lots of little issues than can severely impact running systems if left unchecked. And then when you upgrade one component you'll frequently have to upgrade others to stay within the compatibility support matrix, until eventually you just run the damned SPP to make sure everything in that server is at a "known good compatible" level.

      Sure, we don't just flash as if it were patch tuesday and no-one ever should - we wait for at least 2 months of testing on non-production boxes before we patch any prod kit with firmware unless it's an emergency fix - but lots of people use the HP SPP to automatically download the latest updates; we've had enough problems with them that we'd never do this (and in any case 97% of our servers have no net access). But the whole point of the SPP is meant to be that HP should have already done most of the regression testing for you.

      That said, we've had nothing but trouble with Broadcom NICs for ages and I'm sure there's many admins here who have fond memories of the G6 blades, broadcom NICs, ESX and virtual connect from a few years back. Think HP switched much of their kit to Emulex after that debacle. Also, the latest web-based HP SPP (as opposed to the last one where you just ran a binary) is a complete train wreck on windows for ad-hoc updates, largely due to the interface being handed over to people who seemed to want to make it a User eXperience rather than a tool.

      • Don't try to autodeploy ESXi on broadcom nics.

        Why do you keep blinking RIGHT before you're going to be handed a DHCP address? And shockingly fail 45 seconds later.

      • by cthulhu11 (842924)
        HP's onboard NICs are Broadcom on the G8's.
    • by alen (225700)

      once or twice a year

      what happens is that if you get a bad hard drive or something routine like that HP will make a big deal if your RAID firmware or HD firmware is not up to date or too many versions behind. so i run these service packs once or twice a year

      • by Rich0 (548339)

        once or twice a year

        what happens is that if you get a bad hard drive or something routine like that HP will make a big deal if your RAID firmware or HD firmware is not up to date or too many versions behind. so i run these service packs once or twice a year

        Well, that falls into the category of fixing things if they're broken (if you keep getting RAID failures, check your RAID firmware).

        BUT...

        Why the heck can't they get the firmware right in the first place? I appreciate the value in being able to update the firmware in the event of a rare problem. What I don't get is when the problems aren't rare. I ran into an HP server whose RAID kept failing drives until we updated the firmware on the RAID. For whatever reason the unpaid mdadm guys are able to build a

    • That is a terrible policy. I spent a long night at an office of a fortune 500 company for that very reason. They didn't see any reason to apply bios patches because they were just to add support for newer hardware, not to fix any sort of vulnerability. Fair enough. Several years went by and their terminal server had a processor go finicky on them. They determined the available spares included processors that were compatible. I asked "has the bios been updated to support the newer processors?" I was assured

      • by Rich0 (548339)

        So, the server failure shouldn't be a big deal since everything is redundant in the first place. Then, anybody maintaining servers for which downtime is critical should have the necessary replacement parts just ready to go. They shouldn't be relying on other production servers to do configuration/etc work.

        I don't think the solution to these problems is to just always keep reflashing firmware.

    • If our company is typical, we update firmware only when needed.
      Most servers keep most of the firmware for the entire lifespan.
      An exception would be if the OS is upgraded (but even that is unusual).
      A common exception is the HP ILO controllers. One reason we sometimes update them is that there are real improvements, ad it does not require downtime.
  • You don't flash firmware unless it is for an important issue.

    Or at least not until it has been out quite some time so that other people have done your testing for you.

    • by oodaloop (1229816) on Friday April 25, 2014 @09:37AM (#46840319)

      You don't flash firmware unless it is for an important issue. Or at least not until it has been out quite some time so that other people have done your testing for you.

      Your advice isn't really a general solution if, in order for it to work for anyone, some people must not follow it.

      • by gstoddart (321705)

        Your advice isn't really a general solution if, in order for it to work for anyone, some people must not follow it.

        And companies which choose to make themselves test subjects to allow the rest of us to wait for the dust to settle must live with the consequences.

        In some organizations, they are willing to assume the risk. In other organizations, not so much.

        There will always be companies who fall on the side of bleeding edge, and companies who fall on the side of a lot more caution.

        And for those of us who fa

        • Me, if I have a patch for production, it's going to take me better part of a month to go through the process.

          That's going to limit your nimbleness and opportunity. Obviously that's a business decision that can go either way.

          And, having worked on systems where lives could be at stake, I will stick with a more conservative approach.

          If your architecture will put lives at risk when a bad firmware update for a subset of the servers comes in, then it's too brittle. If safety is paramount, fixing that should be

          • by gstoddart (321705)

            That's going to limit your nimbleness and opportunity.

            Well, in some industries, there is little opportunity in nimbleness. In some, risk is OK and pays off in terms of what that gets you. Depends on what you do, and what the systems are used for.

            If your architecture will put lives at risk when a bad firmware update for a subset of the servers comes in, then it's too brittle.

            Who said anything about architecture? I'm talking about end-to-end systems. If a production system went offline because the NICs we

  • by Movi (1005625) on Friday April 25, 2014 @09:09AM (#46840111)

    Aren’t they also the ones who limit their firmware updates only to customers who have support contracts? I guess you get what you pay for..

    • by jaak (1826046)

      Actually, HP offers free lifetime warranties on a lot of their professional networking gear (this warranty includes support, software upgrades, even includes fans/power supplies and is transferable). All in all, it's a pretty good deal.

  • by Hohlraum (135212) on Friday April 25, 2014 @09:14AM (#46840151) Homepage

    http://slashdot.org/story/14/02/05/0258244/hp-to-charge-for-service-packs-and-firmware-for-out-of-warranty-customers

  • Known for reliable oscillators and calculators, and then they made a line of laser printers that lasted for a while; great engineers behind all that stuff too. Yes, I remember them. How are they doing now post-Carley? (HP's calculators put Rockwell's to shame. I can still remember the Rockwell jingle from over the radio, "big green numbers, and little rubber feet.").

    • by kevmatic (1133523)

      I imagine they're doing fine, working for the company you're talking about, Agilent.

      Make no mistake: The only thing HP has to do with the company that was founded in the 40s is the name. The company that Bill Hewlett and Dave Packard founded still exists, making great stuff. Its just called Agilent. And they still support scopes and multimeters that say HP on them.

      • I remember reading recently that Agilent were planning to split again so those scopes and multimeters will get yet another new badge on them.

  • i got hit by this (Score:4, Interesting)

    by alen (225700) on Friday April 25, 2014 @09:21AM (#46840205)

    didn't brick my server but it screwed up the device list in Windows and caused a cluster not to see the one node where i upgraded the drivers/firmware. put a null device driver into the device manager and i had to delete it and all was OK. just $250 to MS to figure this out since didn't think it was a HP issue

    on the server the network worked and all but the NIC's weren't "seen" by Windows and so the clustering was screwed up

  • Oh, HP... (Score:3, Funny)

    by Greyfox (87712) on Friday April 25, 2014 @09:35AM (#46840297) Homepage Journal
    It's probably a mercy killing. Some of those poor servers were probably forced to run HP/UX. I'd want to die, if I had to run HP/UX...
  • As a network engineer, I can see being involved in arguments between the server platform support teams (read: off-shore) and the network engineering teams (read: on-shore). It'll be like this; "we need network support on a call" "Hello. what's wrong?" "The entire network is down for everyone!!!!! You need to fix this!!!!! The support we get from you is horrible!!!! AAHHHHhhhhhhh!!!!!!" "OK. What changed? What was being done at the time the entire network disappeared for everyone?" "we (15 peo
  • by Hamsterdan (815291) on Friday April 25, 2014 @10:45AM (#46840859)

    Don't manufacturers test their updates? it's not like they couldn't keep some of the stuff they sell for said testing...

    • Yes, but management will ship with known bugs. Development will keep developing until the very end making it impossible for QA to keep up.
  • Is the article suggesting that the Broadcom NICs that HP used in the old Proliants actually _did_ work before this update?

    That goes against years of experience in the field with those things.

  • With HP only one problem persists. Namely, it is what the money cant do.
  • This is one of the first rules of administering servers -- unless it's an absolute necessity, let someone else find these firmware bugs.

    This is especially true now that firmware controls so much in modern hardware. I've had business PCs that have gone through more than 10 EFI revisions in their 18 month lifecycle, and all the release notes show that they fix surprisingly low level things.

    The unfortunate trend is that these firmware bugs are more and more prevalent. It seems like manufacturers are skimping o

  • Their last update bricked broadcom blades of the G1 variety.

Life. Don't talk to me about life. - Marvin the Paranoid Anroid

Working...