IPMI: Hack a Server That Is Turned Off 90
UnderAttack writes "A common joke in infosec is that you can't hack a server that is turned off. You better make sure that the power cord is unplugged, too. Otherwise, you may be exposed via IPMI, a component present on many servers for remote management that can be used to flash firmware, get a remote console and power cycle the server even after the normal power button has been pressed to turn the server off."
Different networks (Score:5, Insightful)
We keep the management network and the production network on separate physical networks. So if you get into a box, you still can't IPMI to any other box.
Also, this is not hacking, it's by design.
Re:Different networks (Score:5, Insightful)
Re:Different networks (Score:5, Insightful)
They will be horrified if they ever run across standard management modules like iLo2 or DRAC.
This "warning" is like the warnings for dihydrogen monoxide and hydric acid.
Re: (Score:3)
Re: (Score:1)
How does this help? On all systems you can connect to the IPMI if you have root on the box. Then some have a built in telnet/ssh client. So not so hard to get into another IPMI instance on another box.
Re: (Score:2)
How does this help? On all systems you can connect to the IPMI if you have root on the box. Then some have a built in telnet/ssh client. So not so hard to get into another IPMI instance on another box.
On shared chassis servers (blades, etc...) you may have common I2C sensor busses, and IPMB... This often includes the ability to send commands to the other blades in the chassis. You can do this even if there's no management networking configured.
Re:Different networks (Score:5, Informative)
On shared chassis servers (blades, etc...) you may have common I2C sensor busses, and IPMB.
Absolutely, remember to ask your vendor *specifically* about this. Various solutions have varying degrees of resilience. Some even treat that topology as *trusted* and made certain design choices based on that.
Surprisingly, IBM BladeCenter is a tad frightening. While the management bus is pretty much isolated from the running OS requiring some hypothetical service processor exploit to get at, the 'CIN' is extended to server processors via VLAN tagging on shared nics. To its credit, they have taken some significant measures to prevent the NICs from behaving in certain ways that seemingly prevent it, but it all seems more complicated than I would like.
Despite BladeCenter frightening me, IBM Flex actually is perhaps the most strongly isolated chassis design. Servers in an enclosure have no i2c or similar connection between them, it's a point to point between the management module and each service processor. The ethernet topology is hard-isolated from the OS. Even if you found a service processor exploit in theory, that service processor has a distinct authentication scheme from everyone else, meaning they can't take the exploit further into other parts of the chassis.
Re:Different networks (Score:5, Interesting)
Cool. but sometimes I hear weird rumors [pcmag.com] about Intel vpro, which make me wonder "what is a network?" If your CPU (?!) is listening for 3G radio signals, there's not just "management network" and "production network" but also "their network" although I guess you can always have your computer wear a tinfoil hat.
Re: (Score:3)
We keep the management network and the production network on separate physical networks. So if you get into a box, you still can't IPMI to any other box.
Also, this is not hacking, it's by design.
Agreed, this is basically a "wow there is a feature, that works by design, and this is what it does!" article. To your comment about different networks, I have found on some servers even though you have the management nic segregated, the service processor goes after the other nics in the machine for IPMI access too. You have to be very careful and never ever trust the default settings on those things.
Re:Different networks (Score:4, Interesting)
We keep the management network and the production network on separate physical networks. So if you get into a box, you still can't IPMI to any other box.
Are you sure? If you have an IPMI management network, that means your server has at least one connection to this network, including a physical Ethernet connection that can reach this management network, and an IP address assigned to its own IPMI service processor.
Who is to say that a hacker can't coopt this server's presence on the IPMI network, and utilize _that_ to gain access to the IPMI management of other servers?
Are you claiming your IPMI LAN is a routed network, where the network infrastructure outside your server guarantees that two different servers can never talk to each other?
Re: (Score:2)
Private VLANs would help this, but only if the function of the Isolated Ports is extended all the way to the hardware and isn't shared at any point (like in a blade chassis).
http://en.wikipedia.org/wiki/Private_VLAN [wikipedia.org]
http://www.youtube.com/watch?v=tbG9YboATvA [youtube.com]
Re: (Score:1)
Who would say such a thing? (Score:5, Insightful)
Saying "you can't hack ..." is just stupid because there's no bigger challenge. That's famous-last-words material right there.
Re: (Score:2)
The article is silly.
Re: (Score:1)
I don't know if there's any equipment that's both that precise and that powerful though...sounds like an interesting challenge.
Maybe add a faraday cage to the list.
Re: (Score:2)
Re: (Score:2)
Change the password (Score:2)
Any sane admin would change the management password when installing the server. And I don't remember any recent vulnerabilities related to ipmi password.
Re: (Score:3)
That's all fine, as long as (a) the person setting up the machine actually knows that the motherboard includes IPMI and understands the significance, and (b) the password changing tools actually work. I was setting up a system with a Supermicro board recently, and the software was quite happy to let me set a reasonably secure password, but then apparently that password was too long for the log-in screen in another part of the system to accept! I was completely locked out. Yes, really. The solution required
Re:Change the password (Score:5, Insightful)
However, IME it's a half-baked technology
It's a very robust and reliable technology. Just because supermicro pushes half-baked tools doesn't mean the technology is flawed. People go with supermicro because they are cheaper than the IBM, Dell, and HP implementations, but you get what you pay for.
Re: (Score:3)
I just use ipmitool locally (on linux) to set up each new machine, and it always works fine (on Supermicro as well). Sometimes slightly different settings are required for different vendors, but the differences are minor and can be done in a script.
Re: (Score:2)
I'm genuinely interested to hear more about this. We went with Supermicro on that particular box because the company building the custom system for us has had generally good experience with Supermicro motherboards in high-end machines. However, their experience with IPMI specifically was limited, so we were all judging by general reputation rather than this issue specifically. We were also all a bit surprised by the poor quailty of Supermicro's IPMI tools, and even more surprised that, once we knew what to
Re:Change the password (Score:5, Informative)
Disclaimer: I'm an IBM employee.
My personal take is that if you are going with supermicro or similar, you best understand how the standard(s) work and use vendor-agnostic tools and steer clear of the tools that SuperMicro and similar provide. For example, understand how IPMI security model works, disable cipher suite 0, refrain from using ipmi 1.5 straight authentication, set user table up so that the first user is effectively disabled, and all other ipmi account slots are all managed securely by your processes. Rather than use setup dialogs and utilities from supermicro, use tools like ipmish from freeipmi and/or ipmitool for one-off usage or to build your own infrastructure. If you want a bigger infrastructure that is a bit more designed from ground-up for mass server management, then something like xCAT can help, but it's not exactly easy to use, might be a bit more ambitious than what you want to do, and despite best efforts it will still be no substitute for understanding the security implications of the implementation provided by the vendor (e.g. right now it only messes with the first 4 user slots, assumes that if cipher suite 0 is there, it's the first one, and so on and so forth).
Now if this seems overly complicated, then buy from a vendor like IBM or HP (Dell I have heard varies more between the spectrum of direct Tyan/SuperMicro and IBM/HP standards, but perhaps closer to IBM and HP for the most part). The standards based tools still continue to work, but the factory default in modern systems is more carefully considered and what proprietary tooling provided is generally more robust than the white box vendors have time/resources to bother with. While these vendors have an increasingly difficult job to distinguish themselves from cheaper competitors from Taiwan, they still do spend some of the money entailed in the price premium on a more finished and confidence inspiring experience.
Keep in mind that as recently as 3 or so years ago, us 'tier one' vendors had pretty crappy defaults and inconsistent tooling as well. However, at least in IBM world, that was all very forcibly corrected for as of the 'M2' generation (the models that released with Nehalem processors).
Re: (Score:2)
In the interest of trying to unfairly disparage competitors, I have been giving Dell and HP the benefit of the doubt until a customer tells me otherwise. It may be the case that the good things I've heard about HP come from people heavily bought into HP's proprietary protocols rather than IPMI.
I will say IBM has had solid IPMI implementation (with some admitted historical sore spots, e325 and e326 in particular).
Re: (Score:1)
Keep in mind that as recently as 3 or so years ago, us 'tier one' vendors had pretty crappy defaults and inconsistent tooling as well. However, at least in IBM world, that was all very forcibly corrected for as of the 'M2' generation (the models that released with Nehalem processors).
If only that were true. I recently demo'd an IBM M3 box, hoping that UEFI would offer salvation from legacy BIOS stupidity, but it just turned out to be more of the same crap with a different name. The box actually displayed a text-graphic during POST directing one to insert a *floppy*, and the serial console is disabled by default. To turn it on, one has to wade through a complex scripting application to generate a bootable .ISO file to be run via PXE or local media. Bloody hell. HP at least has the i
Supermicro (was: Re:Change the password) (Score:1)
Did I miss something that is available for this? I'm asking because we're considering another such system using Sandy Bridge CPUs.
Re: (Score:2)
Sadly, closest IBM comes is x3630 M4. For 3.5", it only gets 14 drives in, and only three pcie slots, or 12 drives and 5 pcie slots. The m3 had 28 2.5" variant, and expecting an m4 refresh along those lines would be reasonable. It seems they make a big deal of going over 2U, and the systems get very very expensive in IBM world, but focus on memory and cpu capacity and actually has less storage capacity.
Re: (Score:1)
Sometimes you don't need the kitchen sink (Score:2)
If you need the features you pay the money, but if you don't why "get what you pay for" when you don't need it?
Of cour
Re: (Score:2)
The point being, if you don't need it, ok. The parent poster indicated that they tried to use the feature, and the vendor provided facility did not work for them.
If you don't need the fetaure, you don't care.
If you think you need remote management, but are not an expert or don't have time to do a platform evaluation, it's safer to go with the 'tier one' vendors.
If you need remote manegement, posess significant expertise on the industry standards and how the vendors might implement them, and have the time a
Re: (Score:2)
IMHO it's better to pay somebody else to evaluate a platform if you don't have the expertise than to just throw money at a vendor and hope.
Re: (Score:3)
You must also disable cipher suite 0. Since it by design makes the password moot and all...
Re: (Score:2)
Any sane admin would change the management password when installing the server. And I don't remember any recent vulnerabilities related to ipmi password.
How about, IPMIv1 authentication requires a fixed key of a specific length, and while any sane admin is not going to leave the Kg_key at 0000000000000000
They may be using the same Kg_key on every server. They may be using the same username and password on every server.
E.g. a compromise of one server may lead to the hacker dumping the contents o
Re: (Score:2)
It's IPMI v2 everywhere, v1.5 on very old servers.
Has this theoretical vulnerability been ever exploited?
BMC? (Score:1)
Seems to me this is more about specific vendor's BMC's used implement IPMI, and the lack of information they provide, and attention sysadmin's pay to securing them.
HP's iLO works for that too. (Score:2)
My solution was to never let a network cable anywhere near the dedicated iLO port. I am not a trusting soul.
Re: (Score:2)
That's nice, but IPMI controllers don't necessarily have their own interface. There's multiple interfaces on the system and you can tie the IPMI unit to one of them, or a serial port.
Re: (Score:2)
Though to be fair, generally the vendor will tie it to the dedicate port, and all other choices require explicit action. So if an admin is local, has no desire for remote management, ignoring the whole thing and leaving the dedicated ports unplugged *generally* suffices.
Re: (Score:2)
IBM servers of yesteryear certainly did no such thing. There IS no dedicated port! PERIOD THE END! There is only a serial port also attached to the system, and two network ports also attached to the system! You can disable an interface that's being used by IPMI at the OS level, but that still doesn't make it a dedicated port.
Systems may NOW have a dedicated management port, I don't know, I haven't seen a server with IPMI in a while, what with the emphasis on clusters.
Re: (Score:2)
Systems may NOW have a dedicated management port, I don't know, I haven't seen a server with IPMI in a while, what with the emphasis on clusters.
What servers do you use now without IPMI?
Re: (Score:2)
What servers do you use now without IPMI?
Cheap ones.
Re: (Score:2)
If the vendor says 'server' it's very likely that it also has service processor capability. E.g. if you get an board capable of hosting a dual intel socket config, the chipset has some bits and pieces that almost certainly allow for remote ipmi access.
Re: (Score:2)
If the vendor says 'server' it's very likely that it also has service processor capability.
Ah, but if I use it as a server, then it's a server. If I don't need a metric assload of horsepower, but I do need another warm CPU someplace, I don't need to spend a lot of money.
It's cheaper to get two machines that don't say server on them than it is to get one machine that does. There was recently a discussion about this on Slashdot. Hot failover is cheaper than redundant power supplies, and something that takes out your power supply just might take out your motherboard anyway.
Re: (Score:2)
Guess it depends on 'clusters'. I assumed high performance clusters where you have more workload than reasonably be accomodated on one or two servers.
If you have a small load with HA clustering for 2-3 servers, it really doesn't matter much what the hardware is nowadays. Given that you want to be resiliant in the face of failures that no expensive hardware redundancy would recover from anyway, it just makes sense.
If, however, you need a few hundred servers, things change. It's not that the management fea
Re: (Score:2)
I usually work for pretty small shops, and people who are trying to pinch pennies. That has its ups and downs, for obvious reasons. But it's a niche I guess.
IPMI and low power (Score:2)
What servers do you use now without IPMI?
Small, low power ones. Supermicro makes an Atom mini-itx board but that is about it. There are many e350 boards which are cheaper and have some advantages over an Atom system but not one of them will operate headless until or unless the OS is running.
Re: (Score:2)
Though to be fair, generally the vendor will tie it to the dedicate port, and all other choices require explicit action. So if an admin is local, has no desire for remote management, ignoring the whole thing and leaving the dedicated ports unplugged *generally* suffices.
Don't try that with a Dell! While I havent worked on their most recent stuff, as of a few years ago their default was to share IPMI on all the network ports. Some IBM servers are the same way. The default addressing might make it unreachable in some scenarios but it is definitely on and listening unless you take steps to make it inaccessible.
Re: (Score:2)
Gets some things wrong, misses some *big* problems (Score:5, Informative)
IPMI is good when used correctly. A quality vendor (e.g. IBM) will select the appropriate security measures for default (*except* still allowing a default password, which really must be changed), but get a random white box, make sure that *this* won't work:
ipmitool -I lanplus -U correctusername -P someincorrectpassword -H youripmidevicehere power off
90% of the time it will. Cipher suite zero is the most idiotic thing in the IPMI spec, it by design allows you to access a system without knowing a password at all.
To Mitigate, ipmitool lan print 1. You'll see what cipher suites are supported and how they are restricted. Usually the supported ones will start out like '0,1,2,3'. In that case, you ipmitool lan set 1 cipher_privs Xaaa' Note the '1' may be different and the '0' in the protocol sequence isn't guaranteed, so adjust accordingly. Also use matching letters if the other cipher suites are not 'a'. xCAT on service processor setup disables cipher suite 0 automatically.
IPMI is actually capable of some very good behaviors if configured correctly.
-In the standard cipher suites, passwords are *never* sent over the wire (encrypted or in the clear) as of 2.0 (in 1.5 this was *usually* the case in sane implementations, but there was a mode for password on the wire). The password is actually a shared secret between client and service processor. In a good client, the service processor cannot be impersonated, as it *must* know the password to respond to the client correctly. ipmitool can be deceived but xCAT cannot.
-Additionally, payload can be AES encrypted as of 2.0 (and usually is, 'cipher suite 3' is the most common incarnation since it has been available since 2.0 of the spec).
-It is pretty much the *only* very specific spec for systems management. If the payload is the sequence of bytes '0,2,0' that means 'turn the server off' no matter who the vendor is. Other specifications tend to fall short of this, in the name of giving the vendors the 'freedom' to do as they wish.
That out of the way, article gets a few things wrong:
eliminate IPMI access over insecure protocols like HTTP. Use HTTPS with proper certificates, or SSH
IPMI cannot be done with HTTP(s) or SSH. IPMI *specifically* has its own UDP protocol in its remote incarnation. Most remote management solutions implementing *also* have a web and cli, but that's not covered in IPMI.
review the BIOS configuration option for IPMI. If you can't have a physical management network, at least try to use a VLAN if supported.
While this is possible in many current shared nic solutions, it generally suffers from two problems:
-In some shared NIC implementations, the OS driver for nic is more likely to mess up the remote access unless it behaves *just* right. This reduces the point of IPMI, to service a system when things have gone horribly wrong. Some older shared nic implementations made it risky even without the complication of tagging, but that's largely no longer an issue.
-It likely doesn't provide significant additional protection over an aliased private IP subnet. The theoretical additional benefit with VLAN being you are protecting service processor access if one server is compromised in-band. The problem is, you can generally coax the shared nic to let the OS onto any tagged vlan (just ask the local BMC what the compromised server's current vlan tag for IPMI traffic is, vconfig add eth0 , and usually you get right on that 'secure' vlan).
try to integrate IPMI authentication with existing authentication systems. Options typically include RADIUS and AD.
Actually, since the *standard* security mechanisms are all shared secret base, this isn't possible. There are some proprietary IPMI cipher suites that have the client transferring a password, but since normally that doesn't happen, you have no user password to check against the authentication server.
The video they showed was a web interface, *not* IPMI. I know companies like SuperMicro have really made this confusion, but I already mentioned that.
IPMI & Security or lack of it (Score:2)
I was working with a large financial institution that could not understand how a large number of servers were shut down when the servers were in a strictly restricted server room. The servers were running VMware's ESX. Looking thru the logs I was able to see evidence that the servers were shut down -- PowerButtonHandler log entry. The IT admin kept on insisting that this was in error as nobody had entered the server room thus nobody could shut down the hosts. It took quite a bit of explaining how IPMI works
Dummies IPMI question... (Score:3)
I know nothing about IPMI so sorry if this is a dumb question:
If it doesn't require the CPU or RAM so be on then am I right in assuming that IPMI is some sort of separate system on a chip (or similar) on the same motherboard as the main computer? Or does it wake the main CPU up and use that?
If it is a SoC or a microcontroller with a few extra bits - has anyone hacked it to do more interesting stuff other than just update server parameters?
Re: (Score:3)
It's a completely separate CPU (BMC - baseboard management controller). It works when the main one doesn't receive any power. I haven't seen any BMC hacks, maybe because they are different for each vendor.
Re:Dummies IPMI question... (Score:4, Informative)
It's a SoC. Each vendor's solution exhibits different levels of restrictions and ambitions.
I recall years ago, one whitebox just gave you root ssh access to the linux system that was the BMC.
On the other hand, most of the time it's very tightly restricted to specifically do things like alerting, power on/off, remote video and serial, event logs, etc. Mostly because the whole point of the stack is to *always* work, and the more complicated you go, the higher the risk gets that you will fare no better than the OS instance on the server you are trying to manage.
IBM and the like will do things like system firmware management and changing things like hyperthreading via service processor, but still steers very clear of allowing open-ended usage of their service processor.
Re: (Score:1)
It's often an ARM SoC, sometimes running straight Linux. SuperMicro has in the past shipped ARM-based IPMI modules running known-vulnerable versions of HTTP servers, etc.
Additionally, the modules can often be reflashed from the host machine itself. If someone gained root access to the host, they could reflash the IPMI module with custom firmware, and then use that to explore the management network itself.
This means that it is important to segment the management network in some way. It should not be possible
Re: (Score:1)
"off" deconstructed (Score:5, Funny)
Once upon a time there was a giant button labelled "off" and "on". Generally one didn't have to think too hard about it's function. Unless you failed to realize that the original IBM PC contained an RC circuit to hinder clickers (if you were too abrupt with the one-finger reset, the switch was ON but the PC wasn't). Tellingly, when the power states were iconified there was no provision for "ON, but only if you waited long enough first". Universal language is not big on drawing distinctions. While the icons were a little more open to interpretation than most people supposed, you could usually find one prominent switch on any device with the semi-scrutable line art, and then toggle for satisfaction.
Personally, it's pretty clear to me that off means off. Off means not doing anything I don't know about, and preferably hardly anything I do know about. A battery-powered RTC falls into the category of things I know about. Beyond that, category membership is extremely limited. I'm already drawing the line if the RTC contains a wake-on-alarm feature which fails to activate an external strobe light when armed.
Perhaps we need a new ITC symbol meaning "not nearly so OFF as you might like to think". The self-evident circle could replaced by a baby Pacman with the missing wedge rotated around to signify a sleeping cap. Less off, but more cute. Baby Pacman dreams of electric sheep while his intestinal flora multiply promiscuously.
Man/woman, dead/alive, off/on. Eternal certitudes, RIP.
Re: (Score:2)
Man/woman, dead/alive
I'm a trans-gender zombie, you insensitive clod!
Aehm, this is obvious? (Score:2)
Why is this worth a story? Are there really people that manage servers but do not even understand the basic functions of IPMI?
Well, I guess there are. But this story will not help them either.
Re: (Score:2)
Of course it can, but why would one want to do it? IPMI isn't available on desktop machines, only on expensive enterprise servers, where most admins really need its functionality.
Re: (Score:3)
I read the stupid article and the entire discussion (didn't you read my previous comments?). IPMI is a great tool and there is no way I'm turning it off in a data center, instead of securing it. Not going to expose it to the internet, of course.
Re: (Score:2)
Actually, badly configured or unconfigured IPMI makes you vulnerable.In some ways, properly configured IPMI makes you less vulnerable.
Bad IPMI is SA fail at SA (Score:1)
Most important part of the sys-/security- admin role is to keep users from shooting themselves in the foot by denying them access to luxuries they don't really need, c.f. sudo - because root gives you everything so you don't NEED the root password. IPMI does not need to double as-/interact with- remote system management.
IPMI becomes most dangerous when SAs fail to apply those same rules and go for convenience features, like IPMI that doesn't have a dedicated port/wire, or IPMI that can talk to the bios.
So c
Re: (Score:1)
Yes, there is some extra inconvenience in accessing the IPMI interfaces of our servers, but each cab has a dedicated physical switch for the IPMI connections which blocks anything but IPMI-UDP packets. Each of our servers either has an IPMI card or a dedicated on-board IPMI NIC - and I don't mean we use one of the CPU-accessible NICs for it. Each cab's IPMI-switch is connected to a console box at the end of the row providing terminal-based IPMI access or you can remote console it.
been saying if for years... (Score:2)
IPMI Surviving vendor infections so far (Score:5, Insightful)
So far IPMI has managed to survive all the crap vendors try to add to it. Chalk one up for standards! It's gone through some growing pains (the original standard was badly broken and precursors tried to share the ethernet port on the mobo), but has settled down into a usable implementation in the last few years, as long as you only use open-source IPMI tools like 'ipmitool'. There are typically still hicups with the video sniffing implementations (particularly when the OS switches video modes), but serial port forwarding seems to work quite well.
In anycase, IPMI has been a godsend for those of us with smaller server installations. It removes the need for an addressable PDU and removes the need for a console server. It removes a lot of cables.
Once the BIOS is setup you don't need to connect a keyboard or monitor to the machine ever again, even when you are physically in the machine room. You just plug you laptop into the switch or Wifi and poof.
There are numerous ways to secure the IPMI net. If configured properly IPMI 2.0 has reasonable security but still can be DOS'd. The best thing about the standard is that it is trivial to insert machines, IP filters, and other things between the IPMI network and the rest of the world but the fact that it can connect directly to the internet also ensures that those extra gadgets are going to be fairly cheap in order to compete. Always only use the IPMI 2.0 UDP protocol... if the thing has a web server or ssh access (easy to test), turn those off straight away.
Mobos with IPMI tend to cost ~$20-$50 more and the IPMI module itself (when not built in) typically costs $30-$60. Considering what you get for it, it's damn cheap. IPMI has already saved me thousands of dollars worth of equipment, gas, and time.
-Matt
The trouble with IPMI is the defaults (Score:2)
IPMI is useful but a huge security hole with some bad default settings. Some servers come with ADMIN/ADMIN as the factory default. Some Dell servers use "admin/calvin". The problem of securely getting a server from in the box to in service wasn't well worked out.
IPMI systems themselves can be vulnerable. Some are small ARM systems running Linux. [serverfault.com] Badly configured Linux. "Older versions of the X8SIL-F IPMI code accepted ssh connections no matter what password was given. The software would then check the p
The only secure computer (Score:1)
"The only secure computer is one that's unplugged, locked in a safe,
and buried 20 feet under the ground in a secret location... and I'm
not even too sure about that one" -- Dennis Huges
Now we just need to create concrete digging computer viruses and we are there.
Skynet (Score:2)
I've been half-joking for years about the fact that most computers (and printers, etc) are never truly "off" these days.
"The power button doesn't actually turn it off," I tell users, "It just tells the device to pretend it's off. Open it up and you'll see lights still on inside." Because you often can. "At least we can still pull the plug, so they can't take over the world just yet...." [nudge, wink, smile] Inside I am not smiling. Because I know that computers (and other electronic devices) increasing
That was quick! (Score:1)
Has somebody woken up from 10 or so years of sleep and found out that there is something like IPMI? Gosh!