IPMI Protocol Vulnerabilities Have Long Shelf Life 62
msm1267 (2804139) writes "If enterprises are indeed moving services off premises and into the cloud, there are four letters those companies' IT organizations should be aware of: IPMI. Short for Intelligent Platform Management Interface, these tiny computers live as an embedded Linux system attached to the motherboards of big servers from vendors such as IBM, Dell and HP. IPMI is used by a Baseboard Management Controller (BMC) to manage Out-of-Band communication, essentially giving admins remote control over servers and devices, including memory, networking capabilities and storage. This is particularly useful for hosting providers and cloud services providers who must manage gear and data in varied locations.
Noted researchers Dan Farmer, creator of the SATAN vulnerability scanner, and HD Moore, creator of Metasploit, have been collaborating on research into the vulnerabilities present in IPMI and BMCs and the picture keeps getting uglier. Last July, Farmer and Moore published some research on the issue based upon work Farmer was doing under a DARPA Cyber Fast Track Grant that uncovered a host of vulnerabilities, and Internet-wide scans for the IPMI protocol conducted by Moore. Farmer released a paper called 'Sold Down the River,' in which he chastises big hardware vendors for ignoring security vulnerabilities and poor configurations that are trivial to find and exploit."
Noted researchers Dan Farmer, creator of the SATAN vulnerability scanner, and HD Moore, creator of Metasploit, have been collaborating on research into the vulnerabilities present in IPMI and BMCs and the picture keeps getting uglier. Last July, Farmer and Moore published some research on the issue based upon work Farmer was doing under a DARPA Cyber Fast Track Grant that uncovered a host of vulnerabilities, and Internet-wide scans for the IPMI protocol conducted by Moore. Farmer released a paper called 'Sold Down the River,' in which he chastises big hardware vendors for ignoring security vulnerabilities and poor configurations that are trivial to find and exploit."
Re: (Score:3)
It seems like every machine I've ever used in a co-lo has had it's drac/ilo card available remotely. That's the point of those cards, after all: so you can get into a machine that has crashed hard and do something to recover it.
Cloud service providers, at least IME, lock down this stuff pretty thoroughly since they only give VMs to customers and so don't need to allow direct access to the management cards from outside the datacenter, and are strict with ports and ACLs even inside the DC. But small compani
Re: (Score:3)
It doesn't have to. Just set up a cheap hardware firewall and keep the IPMI ports on local addresses only accessible via VPN. If the firewall/router can handle the traffic, it's often a good idea to NAT the servers' public interfaces as well so that if you have to change your public IPs all you have to change is the firewall/router config without touching the servers themselves.
Is it Linux's fault ? (Score:1)
Since the IPMI runs Linux, and the researchers are uncovering tons of vulnerabilities on them, would it be correct to say that Linux shouldn't be used on those IPMI, and instead, Microsoft's Windows should be used, instead ?
Re: (Score:1, Troll)
No.
Linux => insecure if incompetently configured
Windows => always insecure
Fortunately, you cannot even get Windows to run on these very small systems or some cretins would surely do what you propose.
Re: (Score:1)
HP have recently released new firmware in May 2014 for their vintage (think DL Gen 5/G1 blades) iLO v1 and v2 after network scans searching for heartbleed vulnerable systems crashed the management processor/BMC. Ah, just reset the device from the OS with the HP windows/linux hponcfg tool, nope, doesn't work as it can no longer talk to the crashed management processor.
The best bit is the fix for these 'remote support' devices, send someone to pull the power cord and wait for residual power to dissipate. For
Re:IPMI vulnz (Score:5, Insightful)
That may be a bit over-the top. IPMI is vulnerable, but it is also useful. Giving IPMI its own physical network with access only after authentication is usually enough. A secure jump-host or firewall that requires authentication to pass traffic does serve well to secure IPMI.
Re: (Score:3)
It's not surprising, and everyone running a datacenter should be aware of this already and run the devices on a separate network isolated from the other networks. Any use of the devices is purely of system administrator interest anyway, which means that it's easy to isolate.
Re: (Score:1)
Many think they are safe just because they used a different IP subnet for the IPMI, until they get screwed by a back-doored switch that can redirect crafted packets from one port to another.
Re: (Score:2)
Good thing IPMI gets some attention. IPMI doesn't seem to be very reliable at all...........
Yes worthy of attention....
The interesting bit is they are built with OLD micro-controllers and designed with OLD economics.
It is clear that modern hobby and educational devices like the Raspberry Pi or Beaglebone Black
shatter the old cost models. Same with the little Chromecast bug with a smaller yet footprint.
It is time to demand updates... and it is also important to know that a little card for very low
budget can do a fine job as a firewall protection resource.... on one side of an inexpensive switch.
S
Common sense (Score:1)
Any function on the motherboard that is connected to the nic, IPMI/EFI/AMT/vPro/etc, are just back doors waiting to be kicked open (if not already opened by default).
Re: Common sense (Score:1)
Re: (Score:1)
That's what they want you to think. Even if IPMI is on a private NIC, everything on the motherboard is ultimately connected to the chipset (c216 connection map [bestofmicro.com]), you'll never know what kind of magic packet any of the back-doored components will respond to because there won't be any logs.
Anything from NIC1/NIC2/PCIe can activate vPro in the chipset, that has built in KVM.
Re: (Score:2)
Sometimes they are, and in that case it's easy to put it all on a separate LAN segment. In other cases it's an odd little setup where BMC and main system share the physical port but have seperate MACs. The BMC gets it's own IP address that the host and it's OS is unaware of. In the bad old days of the mid aughties the the BMC side of the shared port crashed more often than the server. In the worst cases, the host side would crash as well and the only recovery was a hard power cycle ((defeating the point of
Re: (Score:2)
It's not necessarily a choice. Only some systems offer a separate management connection.
That's why IPMI should only live on intranets. (Score:4, Interesting)
Y'know, TCP/IP is inherently insecure. In fact, there's no effective built-in security there. IPMI itself is not secure because the security should not be implemented there; any more than network security should not be implemented in TCP/IP. Since this is a server related issue, IPMI implementers and users are presumed to understand this. Workstation users need not concern themselves with this. What sane workstation user will pay the extra money to get hardware with RAC technology?
Re: (Score:2)
That still doesn't excuse shoddy IPMI implementations and not fixing vulnerabilities when they're discovered.
Re: (Score:2)
Re: (Score:2)
Hey are just following "industry standards". Shoddy, incompetent implementations are the norm basically everywhere, and that includes firewalls, routers, commercial "mainstream" operating systems, etc.
Re: (Score:2)
Anybody see a problem with that?
Re: (Score:2)
Completely agreed. In this case, the vulnerability is deep in the specification for IPMI. They'll need a new spec.
Step one, lose all the channel crap. There should be two. Channel 0 which is accessed via the host OS requiring root/Administrator access. The second is for any sort of remote access (normally LAN). Channel 0 SHOULD skip authentication entirely, channel 1 MAY NOT skip authentication (no NULL user or password). They can take it from there, but I would suggest simple encryption (no, not SSL, I sai
Re: (Score:2)
I will say that the serial channel is useful as well. But this 'all channels are arbitrary' should go. CHannel 0 being the 'in-band, channel 1 always being *the* lan (currently some people have multiple lan channels, this should go away), and channel 2 always being a serial channel, if applicable, could make sense. Usually the serial channel serves as a way to indicate SOL related data and is rarely used for initial purpose of rs232 connected devices, so perhaps reimagine that as just more commands and d
Re: (Score:2)
They have encryption, but it is not mandatory and when used, it is shared secret rather than DH or similar. For the purposes of this discussion, an actual rs-232 connection should be considered remote.
I'm not so sure an SSL cert like system is really practical for target (server) authentication. Management firmware can be expected to be in the field for a long time and never updated. Honestly though, once encryption is established through DH, a simple scheme involving hashing the session key, an arbitrary u
Re: (Score:2)
They have encryption, but it is not mandatory
Same can be said of http and https. Nothing specific to IPMI.
it is shared secret rather than DH or similar.
Well that may be a better way to settle the symmetric key value, but then you have to discuss authentication as a separate item, since Kuid currently serves both in establishing keys as well as authenticating the parties to one another. SNMPv3 USM seems to be a pretty appropriate model for this scenario (where certificate systems are likely to be ignored), which is pretty similar in kind to IPMI except that the client goes first and the key is l
Re: (Score:2)
Well, one IPMI does SHA256 or SHA1. For another, I'm unaware of any attack even against MD5 that would compromise the security when used in an HMAC scheme, as is the case for the hash function use in IPMI.
An actual dump from a BMC:
ID IANA Auth Alg Integrity Alg Confidentiality Alg
0 N/A none none none
1 N/A hmac_sha1 none none
2 N/A hmac_sha1 hmac_sha1_96 none
3 N/A hmac_sha1 hmac_sha1_96 aes_cbc_128
6 N/A hmac_md5 none none
7 N/A hmac_md5 hmac_md5_128 none
8 N/A hmac_md5 hmac_md5_128 aes_cbc_128
11 N/A hmac_md5 md5_128 none
12 N/A none md5_128 aes_cbc_128
As for the rest, yes, http can be done without encryption, but there are substantial low-risk use cases for taht. Http doesn't generally allow rebooting a server into single user mode and resetting the root password..
As for the rest, see A Penetration Tester's guide to IPMI [rapid7.com]. Note that using a DH exchange to negotiate a session key offers forward secrecy and allows for a much more secure authentication protocol that doesn't involve handing out the MD5 hash of any chosen user's passw
Re: (Score:2)
The problem is that it doesn't help unless you implement security on your switches as well (private VLAN or similar). One compromised server can take over the IPMI interface and transmit on the isolated network. This is supposed to be impossible; the host is not supposed to be able to use the IPMI interface to source traffic (assuming it has been assigned a dedicated interface and not shared of course). Unfortunately it is not impossible in practice.
Re: (Score:2)
Sorry to hear your PHB isn't.
Re: (Score:2)
Nobody should ever put IPMI or AMT enabled systems on the public Internet deserves the hacks and system compromises that they deal with. At the *very least* it should be contained within a firewall/VPN on a private LAN or Intranet.
You're doing it wrong. (Score:3)
Design core features.
Write and test/debug core features.
Then, if there is time, do a security audit and glue some security on top otherwise just release what you got.
Security must be built in from step one, step two, etc. and must be and integral part of the design.
Have we not learned our lesson yet?
Re: (Score:2)
New technologies get security audits if there is time? News to me. The money is needed for bonuses for the "manager" that do not even manage to understand the basics.
Most people that make these decisions are not even aware there is a lesson to be learned.
Re: (Score:1)
Then you have never worked for a modern commercial, technical company!
+ *All* benefits go to management, so their incentive is low cost, rapid delivery.
+ Any and all negatives, are laid on the heads of the technical staff, so again
the incentive for management is low cost, rapid delivery.
+ While the technical staff, sometimes, have a different opinion, by definition
nobody cares, since they are "non management". Monkeys make noise? They get the hose.
If by a miracle,
Not protocol vulnerabilities (Score:2)
Bad subject alert: the protocol itself is not vulnerable (any more than any other protocol), the problems are in the implementations (and lack of on-going support for most).
I always set up IPMI on a private VLAN, with only a couple of "trusted" hosts having access. Most things can be done with the "ipmitool" command-line program, or I can port-forward port 80 for the BMCs with a web interface. There are a few web-based BMCs with crappy Java applets for remote KVM (they mangled the VNC protocol just enough
Re: (Score:2)
No, actually the protocol has some ugly parts that make it very difficult to secure, that's why isolation from the internet is the only choice.
I truly despise the funky almost VNC and java app. It makes it quite hard to use it over a secure forwarded port.
Competent people do IPMI over jumphosts (Score:5, Informative)
Even firewalling it is not enough, unless you only let authenticated traffic from very few sources in. In any case, IPMI needs to have its own network segment, anybody putting it into the same segment as other traffic is grossly negligent and utterly incompetent.
And people putting IPMI where it is reachable from the Internet? I think that is grounds for immediate termination with a performance report that makes sure these people never do anything security-related ever again. That level of stupidity is hard to top.
Re: (Score:2)
Even firewalling it is not enough, unless you only let authenticated traffic from very few sources in. In any case, IPMI needs to have its own network segment, anybody putting it into the same segment as other traffic is grossly negligent and utterly incompetent.
The problem is, all you have to do to get complete access to that segment is own either the jumphost or any of the servers with a module on that network. The unpatched vulns in IPMI modules are simply inexcusable. These days, full decent computers are available on a module that size, so they really ought to be able to run a decent OS which gets updates. I vote for Debian. This, finally, is a perfect use for Debian Stable.
Current IPMI stinks (Score:2)
As an IT guy, I really like the concept of IPMI. I would love something like LogMeIn, but that allows you to take control of machines on a baseboard/lights-out level. The only problem is, there aren't any solutions that I'm aware of that offer that kind of easy, useful bulk management of lots of machines from a single pane. But more importantly, the concept of that kind of bulk management should trigger the thought, "Holy crap that opens a dangerous can of worms!" If lights-out management isn't secured
crude fix (Score:1)
Mind you, doing all of this on a BMC might well crash or wedge it into a sullen silence, as they are very easy to DoS into submission even unwittingly (Iâ(TM)ve completely broken BMCs from my testing both Dell and HP servers.)
This sounds like it might be a usable workaround to plug the huge security hole, aim your own DDOS utility at it, definitely using a case of using sparkplug for a lugstud, but if it gets the job done...
am I smoking crack here or were the specification authors?
A common lament on my world.
Trust in Cloud Data Centers? (Score:2)
Given the numbers of attack vectors in data centers from social engineering to software to hardware faults, would you trust your company's IT data system ONLY to "cloud suppliers?"
Something like 2/3rds of small businesses that lose their digital data go out of business within 6 months, so it is a real risk issue to not have a totally local backup data system that can be brought up within a day.
Do you totally trust big cloud data centers?
Warned about this years ago. (Score:3)
I posted about the IPMI threat on Slashdot years ago, after reading the IPMI docs. Now, it's not only a real threat, it's one that's probably being widely exploited.
Even if IPMI packets aren't being accepted from the outside Internet, an IPMI vulnerability means that any break-in to any server allows an easy attack on all servers inside the firewall.
Anyway, for now, if you have a server, do
ipmitool -A NONE -H 1.2.3.4 bmc guid
(replacing 1.2.3.4 with the IP address) and see if it answers. If it responds to that from the outside world, you have a big problem.
Re: (Score:2)
It says
Error: Unable to establish LAN session
Get GUID command failed
Does that means there is no IPMI service, or that there could be one but not with NONE authentication?
Re: (Score:2)
"Unable to establish LAN session" is good. If you can establish an IPMI connection, per Dan Farmer's paper, an attack is likely to succeed.
HP uses a dedicated Ethernet port for IPMI. (Score:2)
Maybe they still have some functionality that allows IPMI over the regular NICs though.
Is IPMI enabled? (Score:2)
Re: (Score:2)
Re: (Score:2)
Worthy of attention, but a tad alarmist... (Score:2)
One thing is that the materials do not distinguish 'service processors' from 'IPMI' the protocol.
The general facets on service processors broadly are no different than any 'appliance': vendors (particularly cheap ones) are lax about security and updates and there is not a lot you can do about it other than pick a vendor that seems to care or isolate the devices. This is nothing unique to the world of 'IPMI'.
In terms of IPMI, there are things in there that should be and in fact are effectively removed by s