Cooling Bags Could Cut Server Cooling Costs By 93% 135
judgecorp writes "UK company Iceotope has launched liquid-cooling technology which it says surpasses what can be done with water or air-cooling and can cut data centre cooling costs by up to 93 percent. Announced at Supercomputing 2009 in Portland, Oregon, the 'modular Liquid-Immersion Cooled Server' technology wraps each server in a cool-bag-like device, which cools components inside a server, rather than cooling the whole data centre, or even a traditional 'hot aisle.' Earlier this year, IBM predicted that in ten years all data centre servers might be water-cooled." Adds reader 1sockchuck, "The Hot Aisle has additional photos and diagrams of the new system."
Yes, but how much does it cost? (Score:3, Insightful)
Re:Yes, but how much does it cost? (Score:4, Funny)
Re: (Score:2, Funny)
Re: (Score:3, Informative)
That's really nifty, and I'm sure it works ok and everything, but... how much does it cost?
Figures cited by Iceotope show that the average air-cooled data centre with around 1000 servers costs around $788,400 (£469,446) to cool over three years. The Iceotope system claims to eliminate the need for CRAC units and chillers by connecting the servers in the synthetic cool bags to a channel of warm water that transfers the heat outside the facility. This so-called “end to end liquid” cooling means that a data centre, fully equipped with Iceotope-cooled servers, could cut cooling costs to just $52,560 - a 93 percent reduction, the company states.
taking the above figures into account as long as the cost to install is under the 200k figure theres an incentive to switch
Re: (Score:1)
Re:Yes, but how much does it cost? (Score:5, Insightful)
The idea that the mainboard components are sealed inside a liquid-filled compartment seems like a major point against the system. Extra proprietary vendor lock-in components mean extra costs of owning and operating, which probably offset any savings from cooling... if any.
I'm skeptical that it will significantly reduce cooling costs (Compared to, say, a chilled cabinet system) because the total cooling load stays the same. If you're generating a billion BTUs of heat you still need to remove a billion BTUs of heat. Any savings will only be from the higher energy densities water allows versus air and maybe initial installation.
Plus, based on their exploded view, there is no less than three heat exchanges before it even gets out of the cabinet: Chip to liquid (via heat sink), submersion liquid to module liquid, module liquid to system liquid. Each time to go through an exchange your temperature gradient goes up.
What they need is a system that is compatible with commodity components to leverage low cost hardware against lower cost cooling. Why not fit water blocks directly to existing mainboard layouts and circulate chilled water from the main loop directly through them via manifolds and pump at each rack? You can still enclose the mainbaord and cooling block in a sealed, insulated compartment to eliminate condensation problems, but not being submerged means you can actually repair/upgrade the modules.
=Smidge=
Re:Yes, but how much does it cost? (Score:4, Interesting)
As for water blocks, I suspect that all the various minor chips in the system would be problematic. Even if your cooling of the CPU, or even the top 3-5 chips, by thermal output, is perfect, there are loads of other components that will heat up and die without airflow. CPU voltage regulators, northbridge, RAM, RAID controllers, ethernet, etc. You can't waterblock them all(at least without a serious redesign that makes using commodity components impossible, or a plumbing scheme that would make Escher wince). Either you go with a hybrid waterblock/conventional air cooling system; which gives you the vices of both; or you have to go with the fluid bath as in this setup.
Re: (Score:2)
Well what I had in mind is a flat plate (say, aluminum) with water channels in it. On this plate there are two or three protrusions that match the main chip locations that need cooling that are milled to physically contact the chips just like discreet heat sinks would.
You attach the mainboard to this plate like you would attach it to the inside of a normal computer case, only backwards. eg; the screws go through from the back side instead of the component side. This puts the components very close to, if not
Re: (Score:2)
Re: (Score:3, Funny)
And mutated sea bass patrolling the aisles to maintain security?
Re: (Score:2)
Re: (Score:2)
I believe that directly cooling components via liquid is way more effective than pushing some air around.
Think air cooled (loud and ineffective) vehicles compared to modern liquid cooled vehicles, that circulate liquid inside the engine (not the combustion chamber of course)...
I agree with the extra cost for the technology, however you could still use the same components if you e.g. submerge things in oil, that does not harm components and does not conduct electricity.
Re: (Score:2)
I don't have time now to RTA but if one was to use say a non conductive, non corrosive refrigerant one could make use of the lower vapor point to more efficantly remove the heat AND even better lower the operating temperature considerably.
Direct boiling is less than ideal because you get hot spots at the bubble nucleation sites. Using a phase change cooling scheme does not have any specific heat removal advantages. All phase change does it guarantee the temperature of the cooling medium - it makes no guarantees on heat flux or temperature of the object being cooled.
And such an environment virtually guarantees non-serviceability of the components.
Hey we run hundreds of hp high voltage motors immersed in refrigerant every day in industrial settings.
Do we now? If you're referring to hermetically sealed compressors (such as in a refrigerator or A
Ugh. (Score:3, Interesting)
For some reason, the filters at work won't let me view the article. Does it happen to mention how much the upfront cost for these bags are?
Re: (Score:2, Insightful)
Like the unpriced bottle of wine at Applebees. If you have to ask...
Re: (Score:2)
It was more of a curiosity thing :-)
I'm wondering about the upfront costs vs. money saved over time after the initial investment.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Applebees has its purpose...the Monterey Grilled Chicken that the location around the corner from my house serves is amazing (although the rest of their menu is indeed questionable.)
Re: (Score:2)
So Applebees does have a purpose - keeping all the lowest common denominator non-foodies out of good restaurants I like. Making a simple sandwich and having it taste ok is nothing I would give a restaurant bonus points for, but I would certainly take points away if they can't even do that right. (I also knew a few people who worked at various Applebees when I was in cu
Re: (Score:2)
Agreed, but the sauce they have on top of it is pretty tasty.
As far as hiring anyone as a line cook, that doesn't bother me nearly as much as my own cooking does...so no biggie :-P
Re: (Score:2)
Re: (Score:2)
I'm the black sheep in my family when it comes to cooking...most of them are great at it, but if it doesn't come in a box I'm pretty much useless.
On the flip side, I am the fastest typer, so I at least have that going for me :-)
Re: (Score:2)
So: sealed l
Re: (Score:2)
Sounds good to me.
I'm still waiting for the day when it is feasable (physically and economically) to lay down small pipes for coolant directly onto a PCB or between PCB layers. That will bring along the true cooling revolution!
Re: (Score:2)
You don't even need to do that... just make a motherboard with a plate behind it that share the same holes for mounting and have a gap for water. Seal it, fill with water, and you have the same thing. If you want to put in transfer "ports" for CPU cooling blocks you can. A motherboard manufacturer could do that now and include a CPU and chipset block with some standard nozzles for connecting GPU block hoses. Drop in a small pump and external heat sink and they could sell it to gamers and server builders
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Not really, things are rather draconian around here...pharmaceutical call center. Oddly enough, Slashdot has always been accessable...It's likely because of someone in IT, lulz.
Excess Heat (Score:1)
TFA mentions using the excess heat to heat the building. I wonder how feasible it would be to actually recycle the heat to generate more power? Anyone have an idea on how much heat could be generated by your typical server farm?
Re:Excess Heat (Score:4, Insightful)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Very little, since you're dealing with very low quality heat. The hottest temp in your system is going to be the hardware itself (unless you're expending energy to pump it - then what's the point of trying to generate power from it?)
So if your max hardware temp is, say, 38C (100F) that's not good enough to generate any appreciable power from.
On the other hand, you probably will be pumping the heat to chill the system, and the rejected heat temp may be quite a bit higher - maybe as high as 75C. You can use t
Re: (Score:3, Funny)
"Very little, since you're dealing with very low quality heat."
And that's why in the really good systems, the only acceptable option is Monster Heat, the finest quality heat available, and its even gold plated.
Do I get at least a pair of rubber gloves? (Score:5, Insightful)
Re:Do I get at least a pair of rubber gloves? (Score:4, Insightful)
You pull that server out of the farm and let other servers pick up the slack while you make repairs.
It's hype, based on the assumption that every server on the planet will be virtualized by 2019, and that the separation of hardware from the software that runs on it will allow IT departments ample time to offload work into "the cloud" while they swap out RAM.
Either that or it's made for large datacenters with multiple redundancies and enormous cooling costs. :)
Re: (Score:1)
Don't want to reduce your smug, but we're doing just this - restart services from the failed component, service the failed resource on a non-critical timeframe. The small shop with a half-dozen server boxes doesn't give a damn about cooling costs or this level of service, for the most part. If they do, they're likely going to someone else to satisfy that requirement, not doing it in house.
I've got stack of servers in my datacenter that are allocatable on demand. Any unused server blade is a potential spa
Re:Do I get at least a pair of rubber gloves? (Score:5, Funny)
What, they won't? Oh man, this virtualization thing is brilliant.
So you virtualize a box, so that, if there's a hardware failure, the box can be brought back up on another machine, with minimal downtime! Also, you can run multiple systems on a box saving money!
We virtualized all our servers around here, went from about 200 servers to 8 machines, each with 16 CPU cores. It went well. So we decided to repeat the process. We then had 4 machines, each taking two VM hosts! It was great, more savings, more vodka for my drawer... So I thought, how could I make this even better...?
That's right, I put all four of THOSE VM hosts on a 486 in the back room that doesn't even need special cooling. Let me tell you, in terms of Vodka, this virtualization thing has been *quite* productive.
How could it not be all pervasive by 2019? I'm sure everyone will be virtualizing all of their VM hosts on VM hosts running on 486s by 2019!
Re: (Score:2)
Re: (Score:2)
Re: (Score:1)
Yeah, I'll keep my FRUs, thanks.
Re: (Score:1)
A few questions (Score:5, Interesting)
Re: (Score:2)
Super cool! (Score:1)
great idea (Score:1)
Re: (Score:2)
we all know what happens when you mix water and server rooms http://www.youtube.com/watch?v=1M_QTBENR1Q [youtube.com] better call up Noah
"The Iceotope approach takes liquid – in the form of an inert synthetic coolant, rather than water – directly down to the component level," the company said.
Re: (Score:2)
Actually, this technology would make the data center better protected from a flood. Since each blade is sealed in its own bubble of coolant, if the entire rack is underwater because of a flood, the blades would be protected. Maybe some of the external components like the cooling pumps might be damaged, but most of the contents of the rack would be fine.
I'm not saying they could continue to operate through the flood, but after the water is gone and the mess cleaned up, you replace the UPS and fix the exter
Coming back full circle (Score:2)
Grandma would be proud of her cold compress technology.
Re:Coming back full circle (Score:4, Funny)
Re: (Score:1)
Re: (Score:2)
Oh, that's listed under hobbies, specifically "cave diving".
Quick Release (Score:5, Informative)
The problem with all this is you need a good piping and plumbing system in place, complete with quick release valves to ensure you can disconnect or connect hardware without having to do a whole bunch piping and water routing in the process. Part of the beauty of racks is you just slide in the computer, screw it in, and plug in the plugs at the back and you're done.
I'm not saying it's impossible, but just building a new case, or blade, or whatever isn't going to do it - you need a new rack system with built in pipes and pumps, and probably a data center with even more plumbing with outlets at the appropriate places to supply each rack with water. This is no small task for trying to retrofit an existing data center.
Not to mention that you have to make sure you have enough pressure to ensure each server is supplied water from the 'source', you cannot just daisy chain computers because the water would get hotter and hotter the further down the chain you go. This means a dual piping system (one for 'cool or room temperature' water and one for 'hot' water). And it means adjusting the pressure to each rack depending on how many computers are in it and such.
The issues of water cooling a data center go WAY beyond the case, which is why nobody has really done it yet - sure, the cost savings are potentially huge, but it's a LOT more complicated that sticking a bunch of servers with fans in racks that can move around and such, and then turning on the A/C. And there is a lot less room for error (as someone else mentioned, what if a leak occurs? or a plumbing joint fails, or whatever. Hell, if a pump fails you could be out a whole rack!).
Re: (Score:3, Interesting)
Re: (Score:1)
Not to mention the very simple fact that when something goes wrong with the servers, you have a team of guys ready to fix it in no time flat.
When something goes wrong with the plumbing no one can touch it unless they're a licensed plumber. He'll take a few days to get there and a few days to do the job, AND he'll charge you more than you paid your server guys in the same time frame.
Re: (Score:2)
I worked for a large investment company some time ago, and we had an "older" data center that was originally designed to house mainframes and used a pool to hold water for cooling. A side benefit of the pool was that employees could use it for swimming, and the water was at quite an agreeable temperature. The benefit here
Re: (Score:2)
The issues of water cooling a data center go WAY beyond the case, which is why nobody has really done it yet . . .
They've only been doing direct water cooling of data center computers since the 1950s. Though the last time I worked on one was in the 1980s, and it was mainframes, not PCs/blades.
Nospill valves and other modern water cooling tech (Score:3, Informative)
No spill (as in "almost insignificant", not as in "not too much, won't empty the whole system, but you better have some towel nearby just in case"), quick disconnect, low resistance valves for watercooled system have been already available for quite some time for enthusiasts.
(Koolance is an example of compagny producing such thing in the US, Aquatuning is an example of shop selling similar implements in the EU - no links to avoid gratuitous advertising to web spiders, but you can easily google the names).
An
Water cooling on that size is no small feat... (Score:4, Interesting)
The ES/9000 that I had contact with was a series of cabinets that were all water-cooled from the outside in...it was a maze of copper pipes all around the edges and back and looked like a fridge. When you opened a cabinet, you could feel a blast of cold air hit you.
It was no trivial feat to do this, they had to install a separate water tank, some generators (I remember one of the operations guys pointing to a Detroit Diesel generator outside in the alley and saying it was just for the computer's water system), moved a bathroom (only water they wanted around the computer was the special chilled stuff), and I can distinctly remember seeing the manuals(!)... 3-inch thick binders with the IBM logo on them, and all they were for was the planning and maintenance of the water system.
No wonder it took almost a year to install the machine.
Re: (Score:2)
It's not trivial as you say but once done (correctly!) can be very flexible. I "managed" (as a systems programmer who had to accept all the designs) a "data center" growing from one water cooled system to several mainframes and to install the "next" system only took two days with everything. Yes, we had extra space / capacity - the capacity plans had 5-10 year estimates (a big fight but paid back later!). DAlso did that for a couple of customer later on.
Liquid (water or other, metals, etc) cooling is more e
Water is a hassle (Score:5, Informative)
I work with particle accelerators that draw enough power that we don't have much choice but to use water cooling, and even though we have major radiation sources, high voltage running across the entire place, liquid helium cooled magnets, high power klystrons that feed microwaves to the accelerator cavities etc... the only thing that typically requires me to place an emergency call during a night shift is still water leaks.
Water is just that much of a hassle around electronics. Even an absolutely minor leak can raise the humidity in a place you really don't want humidity, it evaporates and then condenses on the colder parts of the system where even a single drop can cause a short circuit and fry some piece of equipment. After it absorbs dirt and dust from the surroundings it starts attacking most materials corrosively, which may not be noticed at first but gives sudden unexpected problems after a few years. If you don't keep the cooling system itself in perfect condition valves and taps will start corroding and you get blockages. Maintenance is a pain because you have to power everything down if you want to move just 1 pipe etc...
I just don't see why you would go through the hassle with water cooling unless you actually have to, and quite frankly if your servers draw enough power to force you to use water for cooling then you're doing something weird.
Re: (Score:2)
Re: (Score:2)
The real problem with the system you describe is that a baguette can cause the entire system to overheat. [slashdot.org]
Re: (Score:3, Insightful)
Water (and liquid coolants, even metals) can be a hassle if not deigned correctly. I have had my experiences with water cooled systems but mainly the "over efficiency", well, one burst which shouldn't have happened (LOL).
One thing I have learned (from my son) - in cars, everything replaced with military and/or airplane grade fittings, valves, tubes, etc - makes life much easier. Not much more expensive but very fast pays back. If I would have known that (much) earlier instead of accepting engineering (good
PCs are not quite so fragile (Score:2)
I had my own water cooling experiment about ten years ago. I had a two processor Athlon board and made two aluminium waterblocks for it. Since my metalworking skill was pretty low (and I was limited to hand tools), the blocks leaked, necessitating several patches. First with duct tape (:-), then with plumber's caulk, and finally by covering the whole thing with fiberglass epoxy, which plugged it up. Up to that time I had a nice little waterfall going from the waterblocks down onto the graphics card (a Radeo
Prior Concepts (Score:2)
Cray-2 (Score:4, Insightful)
Hmm... The Cray-2 [wikipedia.org] was cooled via complete immersion in Fluorinert [wikipedia.org] way back in circa 1988. I was an admin on one (Ya, I'm old). So, this is a bit different, but certainly not ground-breaking.
Re: (Score:2)
Oh it's groundbreaking. It's unique. I know, because they have patents on it. 1988 doesn't exist. It's all in your head. They invented something new and innovative so they patented it! Duh.
*cough*
Too early yet for that much sarcasm?
Re: (Score:3, Interesting)
The crays full immersion coolant model hit a big problem - the coanda effect.
This is where layer of fluid near the actual component flows much slower than actual flow - in layers slowing down exponentially as it gets closer to the stationary components.
For air this is not too much of a problem - only a very fine layer of stationary air over compenents that does not affect cooling. But with liquids the effect is both noticable and severely impacts coolant flow over hot surfaces - with some then "next gen" cr
Re: (Score:3, Informative)
The crays full immersion coolant model hit a big problem - the coanda effect.
You did not describe the Coanda effect, you described boundary layer issues. I don't know enough about the story to know whether boundary layers were the real issue: It's pretty routine to take into account the fact that friction causes the fluid velocity to approach zero at the surface of a stationary object, and to account for a lack of turbulence in the laminar part of the boundary layer that reduces heat transfer. It could just be that they needed to get a phase change at the surface in order to pull
Re: (Score:2)
Everything *trickles* from Supercomputers/Mainframes eventually.
Cold mineral oil. (Score:1)
Sixteen years ago, at the end of my highschool career, I was very into overclocking (had multiple celeron 300A). With peltier cooling I was able to run a 300mhz CPU at 450mhz with rock solid stability (ran things like prime95 24hrs a day for weeks). People were starting to experiment with liquid cooling commodity white-box computers.
One of the more interesting applications I saw was an old styrofoam cooler converted into a PC case. All components were submerged in a bath of cold mineral oil. I remember thin
Re: (Score:1)
Re: (Score:2)
How did you get a 300 mhz CPU in 1993?
Doesn't look practical (Score:2)
Look at the cross section photo. This dispenses completely with convection (air flow) and instead designs the system for direct physical contact from the heat sink to the components. Then the water flows behind the heat sink to take the heat away from that.
The problem is that means that you have to make a heat sink with varying height "fingers" on it to meet every component that produces heat (which is all of them), which means every time you change a component you have to redo the heat sink. And of course
Re: (Score:1)
Re: (Score:2)
That's noticeably worse. Component manufacturers haven't tested their components for extended immersion in liquid, even relatively inert ones. This would drive the cost of the device through the roof.
In addition, I know people like to think of heat transfer as radiation or conduction, but convection is the biggest factor, even in a liquid cooled system, this is why the liquid circulates instead of just sits there. And in this case, the liquid is going to just sit there, the area on the motherboard side of t
Re: (Score:2)
That was my thought too.
I can see heat sinks with liquid pipes in them in the future. Plus regular air cooling. ie a hybrid solution.
weight? (Score:5, Interesting)
How much does a rack full of water-cooled blades weigh?
Never thought I'd see the UPS become the lightest thing in the server room.
Re: (Score:2)
Admittedly I've only worked in medium-sized shops, but I've never seen a rack jammed full of batteries.
I've seen the bottom third or so full, and they told me to go any higher would invite stability problems. It's been a few years since I had the discussion, these days do they bolt the racks to the floor and fill 'em with batteries?
Hmmm, so what happens when internals break? (Score:2)
With all those layers it doesnt seem that sliding one of these out and quickly swapping some RAM or any other part is
going to happen.
As well do these Iceotope guys actually make server hardware or just the cooling specs. Who do they get there guts from or are they just advertising and hoping the guys like HP, IBM or SUN (well maybe not SUN) decide to design there next generation of servers with this in mind?
I'd like to see how easy it is for replacement. doesnt look like there is a lot of room for other bit
Re: (Score:2)
Cray XT5 "Jaguar" (Score:1, Informative)
night time freezing of liquid would save more (Score:3, Interesting)
Direct cooling makes far more sense than cooling rooms like I keep seeing around now.
LoB
It is a huge waste. (Score:2)
Either their is huge waste now or they're already expecting to use off-peak energy.
The current situation IS a huge waste.
Basically we're currently trying to cool down server using a fluid which is a *thermal insulator* (air). Of course it's catastrophically inefficient. But it happens to be simpler.
What about the benefits to Joe User? (Score:1)
Almost... (Score:1)
Re: (Score:1)
Maybe we could create a group on Facebook and have fun with our not-quite-working-slashdot-jokes?
I thought we'd finally learned... (Score:2)
Among the many great things the internet has brought us (*cough*porn*cough*), "location-independence" ranks pretty high up there. Your servers don't need to all fit in one cargo container that runs so hot it requires LN cooling. For all it matters, you could put them in a single line of half-racks on a mountain ridge, cooled naturally by the wind (with some care to ke
Re: (Score:2)
OK so then how do you explain this? (Score:2)
"Intel set up a proof-of-concept using 900 production servers in a 1,000 square foot trailer in New Mexico, which it divided into two equal sections using low-cost direct-expansion (DX) air conditioning equipment. Recirculated air was used to cool servers in one half of the facility, while the other used air-side economization, expelling all hot waste air outside the data center, and drawing in exterior air to cool the servers. It ran the experiment over a 10-month period, from Oc
Server standardization... (Score:2)
The problem with this is that it requires server manufacturers to standardize their designs. There was talk a few years ago about standardizing Bladeservers. I don't see this happening as there's too much control in the bladecenter chassis, switch interfaces, management abilities etc. Plus why would IBM want to sell an empty chassis and then let the customer fill it with HP C-Class blades?
Even racks themselves from IBM/HP/Dell/EMC/netapp/Sun aren't standardized, other than they are 19" wide. This is why
Costs of water (Score:2)
The costs of cooling air will be replaced by the costs of obtaining water. This system will not be for "water challenged areas".....Californy, etc.
Done as a hobby 9 years ago (Score:2)
Wow, amazing, they finally produced something like what has been done on my website more than 9 years ago:
http://www.octools.com/index.cgi?caller=articles/submersion/submersion.html [octools.com]
Microfluidics and Home Heat Sinks (Score:2)
I'm surprised that hot chips don't already include a layer of microfluidics right inside the package. People have been dealing with overheating chips inefficiently for years. There's clearly an opportunity to sell chips with fluid cooling built right into them.
I think eventually buildings will have fluid cooling systems attached to heat sinks for all kinds of purposes. Geothermal heat pumps [wikipedia.org] already are popular for making heating and cooling up to 4x as powerful as the electricity powering them (instead of t
how about just cutting down the ac to dc to ac to (Score:2)
how about just cutting down the ac to dc to ac to dc part and make a common DC bus with the big and hot ac to dc part away from the severs and they can just have not as well dc to dc in them.
water has a lot that can go bad with it and do you want some water to mess up a $1000+ sever?
Re: (Score:2)
Going DC doesn't save as much as you might think
There was an apc paper (search for DC on http://www.apc.com/prod_docs/results.cfm?DocType=White%20Paper&Query_Type=10 [apc.com] to find it) on this not long ago.
They considered five systems, three existing and two hypothetical and looked at the total efficiancy including UPS, distribution and PSU in equipment (remember even with DC distribution you still need a PSU and generally said PSU needs to be isolating).
* american AC: 480V/277V three phase from the UPS conver
I spent 4 years doing something similar (Score:2)
I had a start up, Nisvara Inc. 2002 - 2006 We had water cooled and could run whole server rooms with no air conditioning at all! Even had a partnership with NASA Ames.
Our system used sealed copper tube, and something I called a thermal ground, basically a copper or aluminum plate with the tube bonded too it. Then shims that connect the heat sources, the CPU, Northbridge, Southbridge and CPU Power supply and possible ram. Powersupply and hard drives were also connected to the plate to remove the h
Re: (Score:2, Informative)
Ok, so they are British and they spell 'center' with the 'er' the other way around. Why don't they spell server as 'servre'?
First of all, it would be 'serveur', not 'servre'. And its use is too new to be one of those words in which the french spelling is retained from the days of the Normans. Incidentally, even in America, we fail to reverse the 'er/re' in some words, consider 'acre', 'massacre', and 'mediocre', so we're not exactly consistent either...