HP Server Killer Firmware Update On the Loose 100
OffTheLip (636691) writes "According to a Customer Advisory released by HP and reported on by the Channel Register website, a recently released firmware update for the ubiquitous HP Proliant server line could disable the network capability of affected systems. Broadcom NICs in G2-G7 servers are identified as potentially vulnerable. The release date for the firmware was April 18 so expect the number of systems affected to go up. HP has not released the number of systems vulnerable to the update."
Re: (Score:3)
That was available for a week.
Comment removed (Score:5, Interesting)
Re:Hot firmware (Score:5, Funny)
The company budget committee helps us avoid issues like this. The majority of our gear is old enough that there hasn't been a firmware update in four or five years. And no plans to replace it any time soon.
Re:Hot firmware (Score:4, Funny)
They approved two rolls of duct tape for this fiscal year, so we feel well prepared.
Re: (Score:2)
Re: (Score:2)
P.S. I find it sad that bodacious isn't in my browser's or OS' dictionary files.
Re: (Score:3)
I wait until I HAVE TO install a firmware update. Unless there is a major problem that will cause the server to explode, you dont update the firmware.
Re: (Score:2)
Re: (Score:2)
How 'bout it's a major problem because the company won't honor its warranty and support contract until you slavishly install every update?
The good news, though, is that after this update you'll probably have a better reason to open a ticket than the piddly-ass one you had before.
Re: (Score:2)
But didn't you hear? It's on the loose, it could be anywhere and upgrade your servers at any time!
Proliant Nightmare (Score:3)
I deployed a DL380p Gen8 last year, and it gave me heart failure.
Under Red Hat, I needed to change the IP address, so I modified the file /etc/sysconfig/network-scripts/ifcfg-eth0 then did a "service network restart"
Alas, the box did not come up on the new IP. Got to the console which was blank and unresponsive. Power cycled, and the RAID array was GONE (and let's just say this was EXTREMELY inconvenient timing).
Support was able to walk us through some BIOS disk recovery that (thankfully) worked. But I'll n
Re: (Score:2)
Re: (Score:1)
Re: (Score:3)
> I guess a sixer counts as a old-timer now
Nah, just middle-age.
Re: (Score:2)
6 digits? Barely.
And yes, thank you 5 digiters for making me feel young again...
Re: (Score:2)
Get off my lawn!
Re: (Score:2)
Re: (Score:2)
[NO CARRIER]
There's your problem. You flashed your firewall with the firmware package for a mid-90s Hayes-compatible telephone modem. I hope you have a spare; that firewall is hung up good.
If it ain't broke... (Score:5, Insightful)
...don't flash it.
Do admins routinely flash firmware updates in the absence of some identified need? I could see flashing an update if I was suffering from a known problem, or if the vendor identified a security flaw in a previous release. I could see flashing it if necessary to install new hardware.
I just don't see why a server admin would flash a firmware update as if it were Patch Tuesday. In the absence of a security vulnerability or production issue there is no reason to treat a firmware change as an expedited change and not perform full testing before deploying it. That isn't to say that doing some testing of security patches/etc isn't wise - but I can see why it would get rushed.
Re: (Score:3)
and when your entire site goes down on a Monday morning because one of your vendors applied an update to some connecting hardware? And their response when asked for the reason for the outage is "Your hardware was 3yrs out of date. Your Sys Admin said it wasn't broke so he didn't fix it" What's your boss going to say after he gets done telling how many years of your salary the outage cost the company?
I delay updates, but I get that shit approved by executive officers first. I always make sure I have a very g
Re: (Score:3)
and when your entire site goes down on a Monday morning because one of your vendors applied an update to some connecting hardware? And their response when asked for the reason for the outage is "Your hardware was 3yrs out of date. Your Sys Admin said it wasn't broke so he didn't fix it" What's your boss going to say after he gets done telling how many years of your salary the outage cost the company?
I delay updates, but I get that shit approved by executive officers first. I always make sure I have a very good reason to delay it as well.
Ah yes, Sys Admin, the damned if you do and damned if you don't profession.
Re:ITIL (Score:4, Insightful)
Unless the executives don't give you 'non-critical boxes' for every piece of infrastructure to test updates.
"Why do you need an additional SAN at $100k? We'll deal with that if it happens. It happened? It's all your fault!"
Re: (Score:2)
Re: (Score:3)
You know, if that's how your company is being ran, you should already be looking for another job.
Where I work, we've got proper test equipment, a CAB to review the proposed changes, and an expectation that you will test before deploying. When we schedule outages, we have to have a backout plan, and we're expected to have applied the updates in either the lab or a test environment.
The admins aren't considered sacrificial lambs, but they are expected to apply due diligence, test, and identify any risks. But
Re: (Score:2)
Re: (Score:3)
You don't need to be a Fortune 500 company to apply this level of rigor. I'm quite sure we're not one.
Yes, you need resources to do it. Yes, you need corporate will to do it. And you also need to have a company whose culture includes actively assessing risk against their needs, as well as understanding how the risks translate into business risk. If the systems affect the actual production of your business, you need to treat
Re: (Score:2)
I see this all the time, an SMB customer buys telephone or internet service from the company I work for and then the customer thinks we have become their IT department. They are desperate for IT help, but not desperate enough to actually hire someone to take care
Re: (Score:2)
Which is exactly why I do not do ITIL.
Borne from a bunch of technocrats with nothing to do all day long but twiddle their thumbs, ITIL gives any spy organization an exact behaviour profile of your organizations entire operations.
You implement ITIL and your already half way to having your infrastructure compromised.
Re: (Score:2)
And some of the problems we see would have been avoided if the teams responsible had followed the CM process.
The more complex the organization is (we are a truly global company), the more you need structure.
Re: (Score:1)
Structure is fine.
But a canned structure isn't, and no organization benefits from a canned structure it uses that either its competitors use, or its enemies.
Really this has been demonstrated so many times I am quite frankly puzzled why people/organizations feel the need to copy a plan to organize infrastructure, when more than likely they had exactly ZERO input to the process.
Google realized this when it created its infrastructure and came up with thier own plan.
Everyone should do the same, it isn't that ha
Re: (Score:2)
Re: (Score:3)
This is why I believe in half-assing everything. If it's only half done, you can only be half damned either way. Right?
Re: (Score:2)
My boss will be screaming at the vendor, they are not allowed to push any updates until we approve them.
Re: (Score:2)
My boss will be screaming at the vendor, they are not allowed to push any updates until we approve them.
Allowed? hahahaha...
Your vendor has a contractual obligation to... well... follow the contract. The contract does what the contract says... if they screw up, you get money off the bill. It doesn't stop their lead tech from going on a drunken binge after his wife leaves him and taking a golf club to the equipment.
Re: (Score:2)
What moron would sign a contract with a vendor like that, you have drooling morons working as management for your company?
Re:If it ain't broke... (Score:5, Informative)
TBH, I suspect this is just getting publicity since it's the first super-dodgy HP firmware patch since they adopted their "no updates for YOU!" mentality - the explanation for which from HP was that they'd sunk a lot of money into their patching process and people shouldn't get to use it for free I guess. This won't be the last time this happens either.
As a sysadmin that's dealt with dozens of these "killer firmwares", there's often an indentified need. We make extensive use of the HP SPP's at work and they come with a list of fixes and known issues as long as your arm; it's part of my job to go through the advisories to see if we're at risk and if we are to analyse the risk of updating/not updating. Many of them aren't security vulns or emergency fixes and are often extremely obscure, but once in a while you'll encounter something like a NIC locking up on receiving a certain type of packet or the BIOS repeatedly saying a DIMM has failed when it hasn't, or if you mix hard drives with firmware X and firmware Y on RAID controller Z running firmware... er.. A it might drop the whole array... lots of little issues than can severely impact running systems if left unchecked. And then when you upgrade one component you'll frequently have to upgrade others to stay within the compatibility support matrix, until eventually you just run the damned SPP to make sure everything in that server is at a "known good compatible" level.
Sure, we don't just flash as if it were patch tuesday and no-one ever should - we wait for at least 2 months of testing on non-production boxes before we patch any prod kit with firmware unless it's an emergency fix - but lots of people use the HP SPP to automatically download the latest updates; we've had enough problems with them that we'd never do this (and in any case 97% of our servers have no net access). But the whole point of the SPP is meant to be that HP should have already done most of the regression testing for you.
That said, we've had nothing but trouble with Broadcom NICs for ages and I'm sure there's many admins here who have fond memories of the G6 blades, broadcom NICs, ESX and virtual connect from a few years back. Think HP switched much of their kit to Emulex after that debacle. Also, the latest web-based HP SPP (as opposed to the last one where you just ran a binary) is a complete train wreck on windows for ad-hoc updates, largely due to the interface being handed over to people who seemed to want to make it a User eXperience rather than a tool.
Re: (Score:2)
Don't try to autodeploy ESXi on broadcom nics.
Why do you keep blinking RIGHT before you're going to be handed a DHCP address? And shockingly fail 45 seconds later.
Re: (Score:1)
Re: (Score:2)
once or twice a year
what happens is that if you get a bad hard drive or something routine like that HP will make a big deal if your RAID firmware or HD firmware is not up to date or too many versions behind. so i run these service packs once or twice a year
Re: (Score:2)
once or twice a year
what happens is that if you get a bad hard drive or something routine like that HP will make a big deal if your RAID firmware or HD firmware is not up to date or too many versions behind. so i run these service packs once or twice a year
Well, that falls into the category of fixing things if they're broken (if you keep getting RAID failures, check your RAID firmware).
BUT...
Why the heck can't they get the firmware right in the first place? I appreciate the value in being able to update the firmware in the event of a rare problem. What I don't get is when the problems aren't rare. I ran into an HP server whose RAID kept failing drives until we updated the firmware on the RAID. For whatever reason the unpaid mdadm guys are able to build a
Re: (Score:2)
Why don't all those operating system providers get it right in the first place, or car makers, or hell, just about anything?
Operating systems tend to be far more complex than RAID controllers. Generally speaking there is a higher expectation of right-first-time when it comes to hardware/firmware.
Sure, cars have recalls, but they tend to be expensive and rare.
Re: (Score:2)
But plenty of admins would flash to the latest when they get a system to be deployed....
Sure, it only makes sense to start out with a current set of firmware if you haven't started testing the thing yet. If it breaks, I'd call that a warranty issue.
Re: (Score:1)
...that might explain why my iLO locked up recently. It responds to pings, but is completely inaccessible from the web, ssh, and the host OS. Time to call my datacenter and have them power-cycle my server I guess...
Re: (Score:1)
For anyone with ILO2-equipped servers that have frozen due to heartbleed vulnerability scanning, HP has released a (thankfully) free update to the ILO2 firmware to work around the issue. A physical power cycle of each server is required before the update may be applied, however. http://h20566.www2.hp.com/port... [hp.com]
Re: (Score:2)
Heartbleed scanning broke some iLOs by freezing them up, requiring power cycle.
Lovely. I occasionally run vulnerability scans that freeze up printers and the like.
The correct solution to this problem is for vendors to actually correctly implement protocols. The device should accept arbitrary data without locking up, short of that data including a valid password and an instruction to do something that is supposed to cause it to lock up.
Re: (Score:2)
That is a terrible policy. I spent a long night at an office of a fortune 500 company for that very reason. They didn't see any reason to apply bios patches because they were just to add support for newer hardware, not to fix any sort of vulnerability. Fair enough. Several years went by and their terminal server had a processor go finicky on them. They determined the available spares included processors that were compatible. I asked "has the bios been updated to support the newer processors?" I was assured
Re: (Score:2)
So, the server failure shouldn't be a big deal since everything is redundant in the first place. Then, anybody maintaining servers for which downtime is critical should have the necessary replacement parts just ready to go. They shouldn't be relying on other production servers to do configuration/etc work.
I don't think the solution to these problems is to just always keep reflashing firmware.
Re: (Score:2)
Most servers keep most of the firmware for the entire lifespan.
An exception would be if the OS is upgraded (but even that is unusual).
A common exception is the HP ILO controllers. One reason we sometimes update them is that there are real improvements, ad it does not require downtime.
This would be why.. (Score:2)
You don't flash firmware unless it is for an important issue.
Or at least not until it has been out quite some time so that other people have done your testing for you.
Re:This would be why.. (Score:5, Insightful)
You don't flash firmware unless it is for an important issue. Or at least not until it has been out quite some time so that other people have done your testing for you.
Your advice isn't really a general solution if, in order for it to work for anyone, some people must not follow it.
Re: (Score:2)
And companies which choose to make themselves test subjects to allow the rest of us to wait for the dust to settle must live with the consequences.
In some organizations, they are willing to assume the risk. In other organizations, not so much.
There will always be companies who fall on the side of bleeding edge, and companies who fall on the side of a lot more caution.
And for those of us who fa
Re: (Score:2)
Me, if I have a patch for production, it's going to take me better part of a month to go through the process.
That's going to limit your nimbleness and opportunity. Obviously that's a business decision that can go either way.
And, having worked on systems where lives could be at stake, I will stick with a more conservative approach.
If your architecture will put lives at risk when a bad firmware update for a subset of the servers comes in, then it's too brittle. If safety is paramount, fixing that should be
Re: (Score:2)
Well, in some industries, there is little opportunity in nimbleness. In some, risk is OK and pays off in terms of what that gets you. Depends on what you do, and what the systems are used for.
Who said anything about architecture? I'm talking about end-to-end systems. If a production system went offline because the NICs we
Isn’t this one limited anyway? (Score:4, Funny)
Aren’t they also the ones who limit their firmware updates only to customers who have support contracts? I guess you get what you pay for..
Re: (Score:1)
Actually, HP offers free lifetime warranties on a lot of their professional networking gear (this warranty includes support, software upgrades, even includes fans/power supplies and is transferable). All in all, it's a pretty good deal.
Well at least it only affects paying customers ;) (Score:4, Funny)
http://slashdot.org/story/14/02/05/0258244/hp-to-charge-for-service-packs-and-firmware-for-out-of-warranty-customers
Yes, Hewlett Packard. A Genuine Legend. (Score:2)
Known for reliable oscillators and calculators, and then they made a line of laser printers that lasted for a while; great engineers behind all that stuff too. Yes, I remember them. How are they doing now post-Carley? (HP's calculators put Rockwell's to shame. I can still remember the Rockwell jingle from over the radio, "big green numbers, and little rubber feet.").
Re: (Score:3)
I imagine they're doing fine, working for the company you're talking about, Agilent.
Make no mistake: The only thing HP has to do with the company that was founded in the 40s is the name. The company that Bill Hewlett and Dave Packard founded still exists, making great stuff. Its just called Agilent. And they still support scopes and multimeters that say HP on them.
Re: (Score:2)
I remember reading recently that Agilent were planning to split again so those scopes and multimeters will get yet another new badge on them.
i got hit by this (Score:4, Interesting)
didn't brick my server but it screwed up the device list in Windows and caused a cluster not to see the one node where i upgraded the drivers/firmware. put a null device driver into the device manager and i had to delete it and all was OK. just $250 to MS to figure this out since didn't think it was a HP issue
on the server the network worked and all but the NIC's weren't "seen" by Windows and so the clustering was screwed up
Re: (Score:3)
You're running Windows on a server?
The world is a stranger place than one can imagine.
Re: (Score:2)
>don't you think that is a bit silly comment to make?
That was the intent.
Oh, HP... (Score:3, Funny)
Ah, this will help network throughput...... maybe. (Score:2)
Re: (Score:2)
Pardon me, but I've never flashed firmware, but ... "Always have a reasonably quick way to revert if stuff like this happens."? How is this possible on a firmware upgrade?
Testing updates? (Score:3)
Don't manufacturers test their updates? it's not like they couldn't keep some of the stuff they sell for said testing...
Re: (Score:1)
I'm confused now. (Score:2)
Is the article suggesting that the Broadcom NICs that HP used in the old Proliants actually _did_ work before this update?
That goes against years of experience in the field with those things.
what money cant do (Score:1)
Ouch, not good, but not surprising either (Score:2)
This is one of the first rules of administering servers -- unless it's an absolute necessity, let someone else find these firmware bugs.
This is especially true now that firmware controls so much in modern hardware. I've had business PCs that have gone through more than 10 EFI revisions in their 18 month lifecycle, and all the release notes show that they fix surprisingly low level things.
The unfortunate trend is that these firmware bugs are more and more prevalent. It seems like manufacturers are skimping o
Not again! (Score:2)