Supermicro Fails At IPMI, Leaks Admin Passwords 102
drinkypoo writes: Zachary Wikholm of Security Incident Response Team (CARISIRT) has publicly announced a serious failure in IPMI BMC (management controller) security on at least 31,964 public-facing systems with motherboards made by SuperMicro: "Supermicro had created the password file PSBlock in plain text and left it open to the world on port 49152." These BMCs are running Linux 2.6.17 on a Nuvoton WPCM450 chip. An exploit will be rolled into metasploit shortly. There is already a patch available for the affected hardware.
Anyone who trusted SuperMicro... (Score:2, Insightful)
Anyone who trusted SuperMicro for anything business critical gets what they deserved. I had the misfortune of working with their engineering department back in 2006/2007. They were absolutely clueless. Slapping random components together hoping to build good server motherboards, wondering why things would perform oddly or be unstable. They admittedly got it right more often than not, but thats not exactly what you want for servers. Stuff like this is proof they aren't serious business.
Re:Anyone who trusted SuperMicro... (Score:5, Informative)
I manage 10,000 of them. To date lower infant mortality and lower long term failures than I had seen previously with Dell and HP. They also ship a lot faster than Dell or HP. Anyone who exposes their IPMI interfaces to the public internet deserves the results.
Re: (Score:1)
Maybe because they stick to the intel reference designs, which are definitely put together by people who know what they are doing.
HP and Dell get all creative and mess things up.
Re: (Score:2)
Maybe because they stick to the intel reference designs, which are definitely put together by people who know what they are doing.
yeah, but if you want the intel reference design, it's worth it to pay the intel tax. you're going to pay an intel tax when you buy an intel processor anyway. I have literally never had a complaint with an intel motherboard except when it had onboard ATI graphics — Mach64CT, what a POS, you couldn't even trust it to provide a framebuffer without getting the colors wrong.
Re: (Score:2)
I have literally never had a complaint with an intel motherboard except when it had onboard ATI graphics — Mach64CT, what a POS, you couldn't even trust it to provide a framebuffer without getting the colors wrong.
server mainboards, who cares about the colors? That includes windows.
Re: (Score:3)
I have literally never had a complaint with an intel motherboard except when it had onboard ATI graphics — Mach64CT, what a POS, you couldn't even trust it to provide a framebuffer without getting the colors wrong.
server mainboards, who cares about the colors? That includes windows.
But how will you know your Windows server has crashed unless you can see the blue screen?? If its purple or green how will you even know?!?!?
Re: (Score:2)
"Purple?! I didn't know we ran VMware on *that* box!"
Besides, past VMware, everything we ran was Linux, so no BSODs to be seen, just kernel crashes and OOMkillers...
Re: (Score:2)
server mainboards, who cares about the colors? That includes windows.
I didn't say intel server motherboard. I said intel motherboard. These were going to be X terminals, replacing Sun machines.
Re: (Score:2)
But Intel is getting out of the motherboard business? still making small integrated stuff like NUC and Galileo but that's all, unless something is going on with server motherboards specifically.
Re: (Score:2)
But Intel is getting out of the motherboard business?
In the spirit of hope I google'd "intel to continue manufacturing motherboards for servers" and was rewarded instantly with "Intel Denies Report It Will Exit Server Motherboard Business [crn.com] by Rob Wright on June 11, 2013". Short short form, intel has server motherboard products planned through 2015 and their official statement is that they are "looking forward to being the trusted partner to the server channel for many years to come".
I am not always a massive intel fan, but when it comes to motherboards I am pr
Re: (Score:2)
yeah, but if you want the intel reference design, it's worth it to pay the intel tax. you're going to pay an intel tax when you buy an intel processor anyway. I have literally never had a complaint with an intel motherboard except when it had onboard ATI graphics — Mach64CT, what a POS, you couldn't even trust it to provide a framebuffer without getting the colors wrong.
Same here - with the one additional exception being when the motherboard was littered with chips labeled "Intel Experimental**", in which case you kind of expected it to go loopy.
(** ...what? EVERYBODY scrounged the waterfall piles to put extra gear in the cubicle when you needed it. Policy be damned, that's pretty much what the damned things were for.)
Re: (Score:2)
At a prior job, all of the pre-release intel tech boxes we got to preview and test for our purposes were... SuperMicro boxes. That says something to me. At this point in the evening, I'm not sure what, but all those white (well, black was the actual color) boxes were all literally SuperMicro, shipped to us from Intel themselves (with all relevant labels about proprietary blah blah blah).
Re: (Score:2)
I would agree that from most perspectives Supermicro does better than the alternatives (dell etc.), -except- when it comes to IPMI. They are notorious for sucking at this. They really put a lot of effort and attention into making their bmc web interface look awesome and then they half-ass the command line IPMI interface. Every time.
Re: (Score:2)
Supermicro fails, indeed (Score:1, Funny)
They forgot to pay their SCO licensing fee in order to legally use Lunix. Don't forget to pay your $699 licensing fee. Remember, the price goes up to $1399 at the end of July.
Re: (Score:1)
We don't negotiate with terrorists.
Re: (Score:1)
Opportunity for some grey hackery (Score:2, Interesting)
Some intrepid hacker should write a script to take control and apply the patch the vulnerable software.
Re: (Score:2)
Re: (Score:2)
Bad idea. A good idea from the point of logic and security, a bad one from the point of legality. Sadly, the latter is what matters. You go and fix someone's leak and in turn you get to be the bad guy.
Nope. Sorry, but nope. I'm not feeling like being a super-antihero. I don't help people just to get sued for it.
Re:Opportunity for some grey hackery (Score:4, Interesting)
This happened over 10 years ago. In response to the Blaster worm, someone wrote the Welchia worm to find, clean, and patch unpatched machines. Because it downloaded the patch to each machine it infected, its deleterious effects on networks may have been worse than Blaster.
I had the pleasure of being contracted to help remove both worms for a local hospital, sneakernetting the removal tool.
Re: (Score:2)
This happened over 10 years ago. In response to the Blaster worm, someone wrote the Welchia worm to find, clean, and patch unpatched machines. Because it downloaded the patch to each machine it infected, its deleterious effects on networks may have been worse than Blaster.
it seems there is a cheap workaround [slashdot.org]. Whether it will break management is another issue. That's probably still better than being owned. I wouldn't do it either way, of course. That would be ridiculous.
Wha? (Score:2)
It's not super-clear from the article what sort of systems there are, even with the Wikipedia link to IPMI. I mistakenly assumed that BMC was the configuration management company at first...
Without linking to XKCD, can anyone explain this to me like a child?
Re: (Score:3, Funny)
"like a child" ==> Some computers that run websites on the Internet have an "Employees Only" entrance on the side of the building, with a lock controlled by a PIN code (for example, "1234").
SuperMicro built these PIN code locks with the correct code clearly printed on the side of the PIN entry panel.
Re: (Score:3)
SuperMicro built these PIN code locks with the correct code clearly printed on the side of the PIN entry panel.
What's even more frightening is what some of those codes were set to by the security conscious (or is that unconscious) people in charge of them [cari.net]:
[...] at the point of this writing, there are 31,964 systems that have their passwords available on the open market. It gets a bit scarier when you review some of the password statistics. Out of those passwords, 3296 are the default combination. Since I’m not comfortable providing too much password information, I will just say that there exists a subset of this data that either contains or just was “password”.
President Skroob's luggage looks like Fort Knox compared to these things.
Re: (Score:3)
Makes perfect sense why the passwords would suck. These are the same doofus types that put IPMI on the public Internet.
Re: (Score:2)
Thanks.
Who puts their ILO internet-facing? Yikes.
Re: (Score:1)
I *think* they mean that access to the keyboard/video/mouse/power button/cdrom "hardware" emulation is open. BMC would be baseboard management controller in that case, and IPMI is just the spec to access it.
Thus you can remotely login and basically do the same stuff with the server you would if you were physically with the machine. That's pretty terrifying stuff!
Re: (Score:3)
>That's pretty terrifying stuff!
It's pretty handy if you have 100 racks of 30 machines each and no monitor or keyboard on any of them.
Re:Wha? (Score:5, Funny)
>That's pretty terrifying stuff!
It's pretty handy if you have 100 racks of 30 machines each and no monitor or keyboard on any of them.
And with SuperMicro BMCs, it's even more handy when you don't own any of them.
Re: (Score:2)
>That's pretty terrifying stuff!
It's pretty handy if you have 100 racks of 30 machines each and no monitor or keyboard on any of them.
And with SuperMicro BMCs, it's even more handy when you don't own any of them.
And the owner has conveniently wired the management ports to the open internet.
Re: (Score:1)
Or even wired any of them to the local net (which a pro with 100 racks would not ever do, but I have to admit I've done at home), so that a compromise (even mere user-level privs) of any local machine, could then be used to go deeper. Excuse me, I need to go home and unplug a cable.
Re:Wha? (Score:5, Informative)
IPMI is a management interface that allows you to do some neat remote administration tasks on these servers up to and including remote console so you can even install an OS on them over the network. They are a separate network interface with this running. I have several of these boxes deployed in my datacenters and firstly, the IPMI interface is configured with a non-public IP address, and secondly, the box is behind a firewall blocking all traffic that is not explicitly allowed, so while this is some sloppy-ass stuff on Supermicro's part, I am personally not that concerned. I am sure that there are many who are not nearly as cautious as I am though who might need to be concerned. Although if they are also that careless, chances are they might not have bothered to set up the IPMI interface as well or even plugged it in.
hoping my VPN doesn't also have a flaw (Score:3)
> the IPMI interface is configured with a non-public IP address ... so while this is some sloppy-ass stuff on Supermicro's part, I am personally not that concerned.
In my case, those non-public IPs are part of a management network that is only accessible via a VPN. So we're safe UNLESS the VPN endpoint happens to have a flaw, or someone mistakenly plugs one of the management interfaces into the internet, not realizing that the "security" on the interface doesn't actually work.
Re: (Score:2)
On at least some boards the management port can also act as a regular network port.
So unless you take special steps to isolate the management interfaces from each other this sort of bug could easilly turn a single machine compromise into a much larger compromise.
Re: (Score:2)
IPMI is complex remote management implemented by adding a computer inside your computer so that you can compute while you compute, dog. The computer inside your computer is known as a BMC (Baseboard Management Controller).
I have owned precisely one machine with IPMI, an IBM eServer 325 which IIRC is actually made by MSI, it's certainly not made by IBM. The BMC was implemented as a socketed module, in a SODIMM socket or similar but the module was fairly square. That machine had two ethernet interfaces, and y
Re: (Score:2)
It's not super-clear from the article what sort of systems there are, even with the Wikipedia link to IPMI. I mistakenly assumed that BMC was the configuration management company at first...
Without linking to XKCD, can anyone explain this to me like a child?
BMC (the company) does not make configuration management software. They make software designed to torture sysadmins.
Re:Wha? (Score:5, Informative)
In simple language.
It's a VNC connection to the graphics output (and some switches) independent of the main hardware. You can essentially VNC in and reboot the server, adjust bios options, mount a CD from your workstation to the server and install an OS. All while never having to touch the actual server.
It's very handy and a total security nightmare if it's not secured properly which should be obvious from the fact that you can power cycle and have full bios access. As others have said, it should be totally obvious to anyone with any computer literacy that IPMI could be very dangerous.
Re: (Score:2)
They also usually provide serial over lan which allows CLI control of both Linux and Windows (Yes, Windows supports serial console, has since 2003).
Re: (Score:2)
Supermicro boards do have a web gui interface to the BMC (not VNC) , but this is about IPMI which is a CLI to the BMC.
Re:Wha? (Score:5, Informative)
A BMC is a baseboard management controller - it's essentially an always-on processor / chipset that can do basic shit like turn the machine on and off, let you get into BIOS over serial (and thus serial over LAN if your motherboard supports it), etc.
As long as the box has power and the BMC has a connection (typically sharing one of the NICs), you can boot your machine and do shit with IPMI commands remotely, reconfigure the BIOS, whatever.
OEMs build on this by slapping on another layer of shit that lets you do graphical redirection (instead of text), connect over the web, pipe in files and have them emulated as a bootable floppy, disc, or USB image, etc. This lets you do remote BIOS/UEFI/firmware updates for example, a remote OS installation, etc.
DELL calls this shit DRAC or iDRAC, HP has iLO, etc.
Nearly all servers come with a some sort of BMC that supports IPMI. You do not have to pay for the advanced shit that you'll really only ever use once.
When issuing IPMI commands you can require a username and password. You can also enable encryption so that these are not sent in plaintext.
It sounds like TFS is saying that Supermicro had a file containing a list of IPMI passwords in a publicly-accessible space.
Note that if this file just had passwords and not the corresponding encryption keys (RCMP+), they would still be useful. Most implementations make RMCP+ encryption optional - it's on the client to specify the key and keytype used, and its only real purpose is to prevent a MITM from sniffing the username and password.
What moron puts IPMI public facing? (Score:4, Insightful)
What use case? This sort of things should always be behind a firewall. Is it to hard to VPN in? Hell our supermicro IPMI's work rather well though a proxy on the firewall (dell and HP for that matter).
Re:What moron puts IPMI public facing? (Score:4, Insightful)
Exactly. Supermicro definitely screwed the pooch on this one, but so is anyone deploying these systems without a firewall in front of them. It's just common sense.
Re: (Score:2)
Even where you can't use a separate VLAN, you can at least put the IPMI on a different subnet.
Re: (Score:1)
I'm running IPCop on a SuperMicro. It IS the firewall.
Re:What moron puts IPMI public facing? (Score:4, Informative)
Many hosting companies that offer a complimentary IPMI or other KVM-over-IP will give the OOB box an IP address on the public Internet. They do this because it is cheaper than creating a private subnet on a dedicated firewall for each customer and letting them VPN in (like SoftLayer does). I doubt many of these exposed systems are from large corporations that run their own infrastructure, or even cloud providers. They are most likely from the retail hosting business. OVH, Hetzner, etc.
Re: (Score:3)
I was asking about this on the OVH forums just the other day, in fact:
Our IPMI are actually configured on a private network separated from Dedicated Servers network using a private VLAN for all the IPMI traffic fully secured via our network equipement.
There is two way you can access the IPMI connection:
1- Over a Java applet which generate and send you a .jnlp file valid for this session only. (This method let you use keyboard and mouse)
2- Over a webrowser via Serial over LAN that use a temporarly generated user valid for this session only.
https://forum.ovh.us/showthrea... [forum.ovh.us]
Re: (Score:2)
What use case? This sort of things should always be behind a firewall. Is it to hard to VPN in? Hell our supermicro IPMI's work rather well though a proxy on the firewall (dell and HP for that matter).
As with most exploits, they aren't usually easily used unless you have 2 combined.
i.e. someone figures out how to get by your firewall... maybe an employee?
I don't know much about IPMIs though. I do other stuff. So I can't really attest to how exploitable this would be.
A password, stored in plain text ANYWHERE outside of a vault (digital or physical) would be considered a major beach where I'm at.
Arguing that "Well, it was always behind the firewall" would probably lead to you never getting let near sensiti
Re: (Score:2)
Oh Supermicro has plenty of fault. IPMI in general has been vulnerable in many ways since day one, far to many devices would lock up if exposed to general internet noise for to long. Since it's something you do not access often and generally them in an emergency that is a really bad combo. You might know them by the name of a DRAC or ILO card.
Re: (Score:2)
I have seen manufacturers enabling their IPMI implementation by default, sharing the primary network interface, add that many people don't know what IPMI is and you get this problem, lot of IPMI devices accesible from the internet
Re: (Score:2)
It makes a lot of sense to expose it by default for provisioning. However, they need to have a big bold warning that the defaults MUST be changed.
Re:What moron puts IPMI public facing? (Score:4, Insightful)
In increasing order of moron, here are a few ways that this can happen:
1) The IPMI may share the same port as the primary network interface.
2) You may have requested an expensive switching architecture with proper VLAN segregation, but your manager only approved you to take the old D-Link box from under his desk, forcing everything to be on the same segment.
3) The people who run the datacentre may have thoughtfully connected every Ethernet port they could find to your switch, even the one with that funny wrench symbol on it, without telling you. In many cases it's possible for a server to be purchased, received, installed, configured and put into production without any of its owners ever seeing it in person. Throw in a heavy dose of "It's somebody else's problem" all around and anything can happen.
4) In some organizations (and I'm not going to name any), IT policy like "All management ports must be reachable from our head office and the IT support desk in Hyderabad" is set by people who think that "security" means remembering to lock their Lexus.
Re: (Score:2)
#3 for sure.
I know I've been on plenty of projects where the equipment was ordered by one person, physically installed by another, and then the configuration handled by several people, often who don't know each other or have much collaboration. And of course this all is without any deliberate sabotage, hostility, intra-vendor competition or any other skullduggery that might have gone into it.
Each person sees his job as what's exactly on the statement of work and barely has time allocated to do that let alo
Re: (Score:2)
IPMI is awesome for managing servers. All the supermicro mobo's I've ever used had a dedicated ethernet port to make sure the IPMI was on a separate, dedidcated, not-internet connected network. The real problem is that they will (or at least would) fallback to the normal ethernet port for IPMI if the dedicated port was not connected.
So the risk here is anyone who bought nice Supermicro hardware, didn't bother to learn about the IPMI, and only connected the normal ethernet port. It's not going to be a pro
All vendors fail with IPMI v2.0 (Score:4, Interesting)
IPMI v2.0 has a design flaw that any anonymous remote attacker can request and get the salt and password hash for the admin user!
It is a design flaw that cannot be patched.
Better use all of the 20 character allowed maximum password length and rotate the password often!
Re: (Score:2)
This is slightly worse. You can get a plain text list of users and passwords and you don't even have to bother cracking the hash.
Ugh... (Score:3, Informative)
Working on a product based around these now...
As far as I can tell, the Nuvoton WPCM450 is what contains the Matrox G200ew clone for graphics output. Thanks to XAA being discontinued in X.org, the MGA driver is practically unusable for X at this point(even with an ancient, 2d window manager).
Yet another reason to avoid this hardware.
Re: (Score:2)
Working on a product based around these now...
As far as I can tell, the Nuvoton WPCM450 is what contains the Matrox G200ew clone for graphics output. Thanks to XAA being discontinued in X.org, the MGA driver is practically unusable for X at this point(even with an ancient, 2d window manager).
Yet another reason to avoid this hardware.
Blarg? People are using these as servers, not desktops. Given this X support is entirely irrelevant.
Re: (Score:2)
Not always... We need the horsepower for some jobs we're doing, and we have a GUI. Not all 'servers' are locked in racks and hidden away from the world
By default, SuperMicro IPMI attaches to normal eth (Score:5, Informative)
By default, SuperMicro IPMI attaches to normal ethernet. So if you hook up a server to a public connection, you've exposed your IPMI. We caught this in a security audit, we added a dhcp honey pot to our static network to see if we could get any devices to announce themselves. We about shat our pants! There's probably a ton of people at risk not knowing this motherboard is insecure by default!
Re: (Score:3)
The IPMI on my supermicro motherboard only works through one of network ports. In fact it has it's own dedicated port that is only for IPMII (the regular OS doesn't even see it). Though I have seen older motherboards that work like yours I think supermicro has moved in more recent products to dedicated IPMI ports, maybe because of this very reason. You should be configuring the IPMI even if you don't plan to use it, set it an IP and then blackhole that IP on your network. If you don't configure it you don't
Re:By default, SuperMicro IPMI attaches to normal (Score:5, Informative)
By default, SuperMicro IPMI attaches to normal ethernet.
Yes, I saw a mention of that on G+ today, but I lost it. So I went to the source [supermicro.com], I will save y'all the trouble of dicking with the PDF and jump straight to page 2-26 and excerpt the really interesting part:
YE GODS. At least it's in the manual, which no one reads. You can select a port once you've got the system up and running, and once you do that it will stick, but until then it operates unsafely, as above. And if by chance there's no link on the management port during boot, perhaps because the management switch is also being cycled, then IPMI will appear on another interface.
There's no excuse for not firewalling that off, but it's still also unacceptable behavior.
Re: (Score:2)
By default, SuperMicro IPMI attaches to normal ethernet. So if you hook up a server to a public connection, you've exposed your IPMI. We caught this in a security audit, we added a dhcp honey pot to our static network to see if we could get any devices to announce themselves. We about shat our pants! There's probably a ton of people at risk not knowing this motherboard is insecure by default!
Dedicated v. shared ethernet is going to vary per model and some models actually don't support shared ethernet.
Also even when using shared, that is only a danger when a DHCP server is present and the machines are being admined by a idiot who does not know what they are using.
Did anyone read the article before posting (Score:1)
Re: (Score:2)
Did anyone read the article before posting
Yes, I read the entire article. Well, read all of it, and skimmed the rest to make sure it said all the things. You're welcome.
In the article, SuperMicro has fix in update.
Yes, it is mentioned in the summary as well. More information is available at the foot of TFA.
However, the key takeaway is that thousands of people decided to not patch their SH1T.
Yes, one of the things that I found interesting about the article (but decided not to add to the submission, favoring conciseness in pursuit of clarity) was the complaint that "flashing a system is not always a possibility" which I found to be a load of hot cockery. As I said while discussi [google.com]
Re: (Score:1)
Re: (Score:1)
user error (Score:2)
it's crazy to expose IPMI to the public net. yes, that might mean you need separate wiring for an internal subnet, and you might not be able to use all your ports for public access - just read the docs before you buy it.
IPMI crashes (Score:1)
49152? (Score:1)
Port 49152? Uh oh. A Commodore 64 could hack that!
Saved by crappy software! (Score:2)
Ah, well. The only one of my SuperMicro boxes that had a public-facing IPMI address can't be reached; the IPMI software is borked and won't let me assign an IP address. It will take a 200km drive followed by a hard power cycle to get the IPMI up and running again.
A permanent workaround (Score:1)
I found what appears to be a good permanent workaround from a Christian Hertel in the comment section of http://threatpost.com/plaintex... [threatpost.com]:
/nv/ipctrl/rultbl.sav
Another Hotfix in case there is no newer IPMI firmware release to upgrade to (so no way to fix the issue otherwise):
Login via SSH, then issue the following commands:
shell sh
iptables -I INPUT -p tcp --dport 49152 -j DROP
iptables-save >
I've tested it on my affected servers and have verified it works and survives a reboot of IPMI.
Works on Supermicro x9 boards (Score:2)
Home FreeNAS box...
~ # telnet 192.168.7.7 49152 /PSBlock
Trying 192.168.7.7...
Connected to 192.168.7.7.
Escape character is '^]'.
GET
adminADMINmypasswordTTmyuseridmypassword4