Microsoft is Now Submerging Servers Into Liquid Baths (theverge.com) 82
Microsoft is starting to submerge its servers in liquid to improve their performance and energy efficiency. A rack of servers is now being used for production loads in what looks like a liquid bath. From a report: This immersion process has existed in the industry for a few years now, but Microsoft claims it's "the first cloud provider that is running two-phase immersion cooling in a production environment." The cooling works by completely submerging server racks in a specially designed non-conductive fluid. The fluorocarbon-based liquid works by removing heat as it directly hits components and the fluid reaches a lower boiling point (122 degrees Fahrenheit or 50 degrees Celsius) to condense and fall back into the bath as a raining liquid. This creates a closed-loop cooling system, reducing costs as no energy is needed to move the liquid around the tank, and no chiller is needed for the condenser either. "It's essentially a bath tub," explains Christian Belady, vice president of Microsoft's data center advanced development group, in an interview with The Verge. "The rack will lie down inside that bath tub, and what you'll see is boiling just like you'd see boiling in your pot. The boiling in your pot is at 100 degrees Celsius, and in this case it's at 50 degrees Celsius."
Cray. (Score:4, Insightful)
The cooling works by completely submerging server racks in a specially designed non-conductive fluid.
Can someone see if Seymour Cray is still around?
Re: Cray. (Score:2)
IBM was doing that even before Cray.
Re: (Score:2)
Indeed, tons of people have done this before and all stopped doing it as soon as they could because it's a huge pain in the ass.
Re: Cray. (Score:4, Funny)
Indeed, tons of people have done this before and all stopped doing it as soon as they could because it's a huge pain in the ass.
Yep and we all know that once you try something once that you *never* try it again. After all nothing at all has changed in the last 50 years, so you're absurd to retry a concept.
Re: Cray. (Score:1)
Who said anything about not trying things? All I said was everyone who has tried it has found it to be a huge pain and moved on when they could. This thing's cool, but It's a tiny demonstration unit. We'll see if they've solved the problems at scale.
Re: (Score:3)
Re: (Score:2)
Hmmm maybe you're onto something. If MS replaces the fluorocarbon based fluid with something conductive this may actually be achievable...
Re: (Score:2)
Well, the old Crays and such used refrigerant and had pumps which needed maintenance. In this case it's just a tank full of liquid and all the motion is provided by it's own convection heat, so way simpler and less hassle.
I imagine this will be like their experiments on the sea floor where the racks are built with as much redundancy as possible so they don't have to be serviced physically even with a hard failure.
Re: (Score:2)
Re: (Score:2)
I worked on a high freq system which tried to beat other market participant. The code was optimized to hell, there was CPU affinity, everything cut down. Short fibre cables, fastest aruba switches we could find, the list goes on. Then we realized we could squeeze a little bit more out of the CPU with immersion cooling, so, put it in to a tank of mineral oil. It worked well, but NOBODY in the team wanted to do maintenance or hardware upgrades on that server.
Yes, the server was colo'ed in the exchange too. Ca
Re: Cray. (Score:3)
IBM was doing that even before Cray.
Maybe; but IBM didnâ(TM)t have those cute little plastic fishies floating around!
Re: (Score:2)
yes we have been submerging hot electronics in various liquids for decades maybe even a century or more if you look at non computing use applications, but "the first cloud provider that is running two-phase immersion cooling in a production environment."
So its easy to be first when you subdivide it to your specific application
Re: (Score:1)
96.8% of all statistics are meaningless.
Also, BECAUSE CLOUD! It's automatically exciting and interesting and "new".
I saw a demo of a similar immersion cooling bath at a tech convention a few years ago. Initial thought was hmm, nifty, but wow what a PITA. Now it's a PaaS (PITA-as-a-Service) I suppose? I mean, there's some valid use case if you can get true scale on this - to the point where you don't generally repair failed components, just exclude them until batch maintenance/replacement time.
Plus they
Re: (Score:3)
This doesn't sound much like overclockers putting their build in mineral oil. They're using some kind of fancypants fluorocarbon, which sounds vaguely similar to Freon. Using an evaporative refrigerant in direct contact with the parts would function differently than just submerging it in a liquid heatsink like oil. Which is also what the "non-computing applications" you're referring to do, some of them having to circulate it out to a radiator when simple heat absorption doesn't cut it. The Microsoft baths d
Re: (Score:1)
The problem is bath size and failure rate of components. A bigger bath is simpler but in a failure state more equipment must be taken off line, drained, then repaired, and then dunked to go back on line. Smaller baths with less equipment become more complex to manage.
Though you can have small equipment in racks, as long as an machine can access the small unit, pull it, bring it to the surface and take another you down and insert it.
The entire facility could be one big bath and you control an robot picker
Re: (Score:2)
Re: (Score:2)
My first thought as well. Cray-2 anyone?
fluorocarbon-based liquid (Score:5, Insightful)
Re: (Score:2)
what you'll see is boiling just like you'd see boiling in your pot
They have looked into it and have already crafted statements to deflect.
Re: fluorocarbon-based liquid (Score:5, Informative)
Re: (Score:1, Funny)
Re: fluorocarbon-based liquid (Score:3)
Re: (Score:2)
Wait, I think someone found your machine!
https://www.reddit.com/r/techs... [reddit.com]
Re: (Score:3)
Could simply be Fluorinert [wikipedia.org], which has been around since (at least) the mid 1980s -- it was used in the Cray-2, (I was an administer on one at NASA LaRC back then).
Re: (Score:2)
Re: (Score:2)
My recollection is that it is exceptionally expensive, many hundreds per liter.
I seem to remember that too, but that was in 1988-92. Still, it's well-proven tech.
Fluorocarbon liquid will destroy electronics? (Score:2)
I'm guessing that the vapor will slowly destroy electronic components, also.
Re:fluorocarbon-based liquid (Score:4, Interesting)
I watched a youtube video awhile ago about Micky$oft testing underwater datacenters. From the video, after two years underwater the experiment was dubbed a roaring success. Whatever happened to this plan?
Re: (Score:2)
I suspect they will combine the two technologies. Immerse the servers in a fluorocarbon liquid, inside a metal cask that is then submerged into the ocean. Passive cooling perfection. Until the cask ruptures and kills all sea life within 100 miles.
Re: (Score:3)
These are parallel tracks that they're investigating to reduce energy and (once-through, open loop) water usage. One thing they learned from the underwater stuff was that, if you purge the container of oxygen and humidity (probably with dry nitrogen), you end up with many fewer hardware failure. The article points out that this two-phase cooling technology does much the same thing - it
Re: (Score:2)
So, what fucking moron thought a legitimate question was a troll? Come on idiot fess up.
Re: (Score:2)
Specific classes of fluorocarbons (PFAS's and certain HFCs/CFCs) are what are dangerous and bad for the environment. In general, a covalently bound C-F bond is very stable and non-reactive, hence PTFE (Teflon).
If everything is closed cycle, there's no environmental concern. When discarding the liquid, there are dedicated removal services that know how to break down fluorocarbons in an environmentally safe way.
Re: (Score:2)
Google Novec 1230. Seems quite safe and global warming potential similar to CO2.
Will they need DC power to run more systems in one (Score:1)
Will they need DC power to run more systems in one rack? They can pack them in but the AMP limit what can be on each cable / bus.
Also do want 220+ AC to be near this fluid?
What about the fire test?
Re: (Score:2)
Something that is already bonded to fluorine won't be bothered reacting with oxygen. It might even work like Halon and interrupt combustion reactions but that may not work with the coolant formula.
Re: (Score:2)
which is scarier, 10 amps of 220 volt AC or 200 amps of 12 vold DC
one will sting and burn, the other will be like getting hit by a bus, choose wisely
Re: (Score:2)
Grab a large car battery's terminals, one in each hand. You've now got a little over 12 volts potential across your body, from a battery that can easily deliver 200 amps. You know what you'll feel? Nothing. Not even if your hands are wet.
Now try the same with a 220VAC source current-limited to 10amps. Wet your hands and let the ER doc report on the results.
Re: (Score:3)
So, yes, a 12V source is generally safe to touch but comes with a big *
If your hands are wet with something conductive - sweat being a good example - you CAN get a decent zap from 12v if the current potential is sufficient. Same if you accidentally get pierced by some stray, fine wire ends (who hasn't been stabbed by a stray copper strand, really?) or some other way allow the current to get through your skin. Testing 9v batteries demonstrates pretty clearly that it's enough potential to shock you.
Re: (Score:3)
Grab a large car battery's terminals, one in each hand. You've now got a little over 12 volts potential across your body, from a battery that can easily deliver 200 amps. You know what you'll feel? Nothing. Not even if your hands are wet.
Now try the same with a 220VAC source current-limited to 10amps. Wet your hands and let the ER doc report on the results.
Don't do this ever. Use one wet hand to touch both post. You really will get hurt.
You ever drop a wrench on a car battery or bare frame to the 12+ and short the posts of a battery? Ya, instantly welds it to the post. With the human body its more about resistance and path it has to take. Up one arm across the chest and down the other. Too much resistance to really do much. Compare licking a 9v to just touching the posts. You have to press real hard to get conductivity with your hands. Either
Re: (Score:2)
Re: (Score:2)
Ground that AC in a path which crosses your heart and your heart stops. That's how the electric chair works - it disrupts the cardiac rhythm with a 60-cycle per second sine wave the heart muscle can't possibly synch to.
200 amps of 12 volts DC will burn you to a cinder if you actually get it to happen in the first place and don't promptly let go. Getting 200 amps to conduct across the human body at 12 volts DC is rather challenging though. 12 volts is not a lot of "potential difference." You need a very good
Re: (Score:2)
Also do want 220+ AC to be near this fluid?
What about the fire test?
From TFS/A:
If it's Fluorinert [wikipedia.org], which was used literally in the Cray-2 (which I administered at NASA LaRC back in the late 80s / early 90s), it's "electrically insulating". There are videos of people submerging running Macs and illuminated incandescent lamps into the fluid with their bar hands -- and un/screwing in the light bulbs.
According to several spec sheets (here's one [zeiss.com]), Fluorinert isn't flammable:
5.1 FLAMMABLE PROPERTIES
- Autoignition temperature: Not Applicable
- Flash Point: No flash point
- Flammable Limits(LEL): Not Applicable
- Flammable Limits(UEL): Not Applicable
5.2 EXTINGUISHING MEDIA
- Material will not burn.
What are the fluid effects on workers? (Score:2, Insightful)
They will be heavily exposed (like the person in the photograph) making skin and respiratory contact inevitable.
Re: (Score:2)
Perfluorocarbons have been around for many years. They mostly have no long term implications with their most serious issue being short term skin / eye irritation.
There isn't anything amazing or new being done here.
Re: (Score:2)
Yeah, that's why they narrowed the claim down to "the first cloud provider that is running two-phase immersion cooling in a production environment" - it's been done before outside of cloud, and in cloud in non-production, and even cloud in production but not using two-phase cooling. It doesn't mean much.
"Dive, dive!" for servers immersed in water (Score:1)
I can see the devops team saying "Dive, dive!" and the Microsoft Windows startup chime playing...
https://www.youtube.com/watch?... [youtube.com]
Indoor plumbing is the indication of civilizati (Score:3, Informative)
There have been liquid cooled rack solutions for standard servers for years now. Water/Ethanol Glycol is cheap and easily disposed of and deliverable at -30 C. I worked at a data center right across the street from a million gallon tank of stuff held to handle peak chilled water requirement of a city center and convention hall. The data center moved in when it was realized the convention hall had no intent of ever hitting a occupancy rate where the cooling would be needed.
We did not enjoy the costs of full rack solutions solely because of the plumbing contractor costs so we placed coolers at bottom of difficult racks to cool and had a large amount of insulated flex running under the data center floor. The 100+ air handlers were equipped with 2 chilling loops, one entirely internal and one with the utility chilled water. When the costs were figured out, in this very unique case, utility provided chilled water to dedicated air handlers was both more reliable and more cost effective than any other method.
The Utility had less than 20 hours of not delivering chilled water at the contracted temp over a decade, all during scheduled intervals in spring and fall. We had 20 days of the private chillers being out of service for all sorts of issues, including squires and birds. All this was next to lake and a century old infrastructure that had all the 4C year round water that the EPA would allow pumped, that they did not allowed pump citing environmental concerns of a lake next to dozens of cities.
Re: Indoor plumbing is the indication of civilizat (Score:2)
Re: (Score:2)
had all the 4C year round water that the EPA would allow pumped, that they did not allowed pump citing environmental concerns of a lake next to dozens of cities.
But somebody has to think of the wildlife! [tampaelectric.com]
Re: (Score:1)
including squires and birds
Queue Billy's "In the Night"
A few years (Score:5, Insightful)
This immersion process has existed in the industry for a few years now
If by a few years you mean several decades. I hear ghosts of IBM, Cray, SGI, etc calling.
Re: (Score:2)
Generally speaking, immersion is not what they do, they usually use pumps and pipes and keep liquid separate from the boards. While immersion cooling is neat, it generally is impractical to actually roll out at scale (so expensive you'll never recoup investment, and it's much harder to replace a bad part).
component replacement? (Score:2)
So... what if they need to replace a component on the submerged rack? Do they pull it out, then have to "dry" it off before working with it? Or can they work on it while it's dripping with fluorocarbons?
Re: (Score:2)
At room temp, the vapor pressure is 10 times that of water. So it will evaporate very quickly.
Re: (Score:2)
Re: (Score:2)
They probably mark the component bad and leave it until the rack requires retirement.
Re: (Score:2)
Do you replace hardware in a cloud facility, or power it off and wait for the parent module to come up for recycling?
I genuinely don't know. With the submersible data center I understand it was zero intrusion for the entire life of the project.
Re: (Score:2)
This was a philosophy trotted out early. In general, you don't hear it as much because most companies that tried that grew out of it, because it's supremely wasteful. They used this mainly to justify having hardware that would fail without any way of knowing how it failed, but decided it was worth a couple dollars a server to actually bother having the stuff to know how to fix it. There were some very happy 'salvage' companies that were ecstatic when a major computer consumer went to this model, because t
Certs needed (Score:5, Funny)
Man, now I'm gonna have to get SCUBA certified just to be an IT guy.
Boiling coolant? (Score:2)
I can see how this would reduce cost, but:
- doesn't this create a potential issue because the liquid has a much higher heat capacity than the gas bubbles, so you get uneven cooling?
- bubbles produce cavitation, so you get lots of vibration in the liquid.
Re: (Score:3)
The Cray that was cooled this way bubbled like crazy, and it seemed happy. All the liquid needs to do is transport heat away from the board/chips, which it seemed to do. I'm not sure how practical it was in real life, but it sure looked cool at a supercomputing trade show!
Re: (Score:2)
I can see how this would reduce cost, but:
- doesn't this create a potential issue because the liquid has a much higher heat capacity than the gas bubbles, so you get uneven cooling?
- bubbles produce cavitation, so you get lots of vibration in the liquid.
Heat of vaporization is typically much greater than that of incremental temperature increases. So it's possible for this fluid that the greater amount of heat absorbed by boiling more than makes up for the loss of conductive heat transfer into the gas bubbles, so long as there is enough movement in the fluid to carry away the bubbles.
Re: (Score:3)
The gas bubbles are the cooling liquid that's gotten hot enough to undergo a phase change. They're carrying away more heat than the liquid itself can.
Bubbles are a symptom of cavitation, not a cause. Cavitation happens when something, usually a mechanical disturbance, reduces the pressure of the fluid to below the vapour pressure. That's not happening here.
This ... allows Microsoft to more tightly pack H/W (Score:2)
This tub of servers also allows Microsoft to more tightly pack hardware together, which should reduce the amount of space needed in the long term compared to traditional air cooling.
That sounds like these cards were explicitly designed for immersion-style liquid cooling.
Flourocarbons? NOT GOOD. Greenhouse gas (Score:2)
the entire data center (Score:2)
https://news.microsoft.com/inn... [microsoft.com]
Also, I've seen oil submerged motherboards for at least a decade. I remember overclockers did it for cooling.
https://news.microsoft.com/inn... [microsoft.com]
Old News (Score:2)
Old news for goodness sake.
A company here in Perth, Down Under Geo Solutions, has a computing cluster submerged in fluid for a number of years. It's used for heavy data processing for geological processing for resource and petrochemical industries.
This type of use has been around for decades.
Water is weird - thermal capacity (Score:2)
Not really a closed system, right? (Score:2)
At the risk of being Captain Pedantic... The claim that this is a closed system isn't quite true, right? Sure, the liquid boils and then condenses back to liquid once it cools.
For this to happen, however, the system has to lose heat. Excess heat must be radiated out to the surrounding environment.
Re: (Score:1)