Inside the Lucasfilm datacenter 137
passthecrackpipe writes "Where can you find a (rhetorical) 11.38 petabits per second bandwidth? It appears to be inside the Lucasfilm Datacenter. At least, that is the headline figure mentioned in this report on a tour of the datacenter. The story is a bit light on the down-and-dirty details, but mentions a 10 gig ethernet backbone (adding up the bandwidth of a load of network connections seems to be how they derived the 11.38 petabits p/s figure. In that case, I have a 45 gig network at home.) Power utilization is a key differentiator when buying hardware, a "legacy" cycle of a couple of months, and 300TB of storage in a 10.000 square foot datacenter. To me, the story comes across as somewhat hyped up — "look at us, we have a large datacenter" kind of thing, "look how cool we are". Over the last couple of years, I have been in many datacenters, for banks, pharma and large enterprise to name a few, that have somewhat larger and more complex setups."
Rendering (Score:2, Funny)
300tb (Score:1, Insightful)
Re: (Score:2)
By mid-year, my pre-production lab will have 150TB. Our production datacenter, just for PLM alone, has something like half a petabyte.
Am I the only who finds this funny... (Score:1)
I find it funny that Slashdot... (Score:5, Funny)
Let's break this down submission down..
"Hi. I found this article on the web that totally didn't impress me, I think they fiddled with the numbers to make themselves look better than they are, and overall I really couldn't give a shite."
Yes. Obvious front page material for a Sunday!
Hmm? (Score:1)
Re: (Score:1)
Re: (Score:1, Interesting)
WTF? Jabba the blob is not impressed, why would they do that? Wouldn't it make more sense to store metadata in the db and the actual image data on XFS RAID?
Re: (Score:1)
Re: (Score:1)
If they were cattle, it'd be a render ranch. Render farm implies a soil metaphor to me. Every machine I add to my render farm is arable soil, waiting to be planted with precious digital seed to yield my crop of rendered data. A ranching metaphor is much harder to construct.
Re: (Score:1)
Re:Hmm? (Score:5, Funny)
Re: (Score:2, Funny)
Re: (Score:1)
Re: (Score:2, Funny)
That's really not that large (Score:3, Interesting)
Re:That's really not that large (Score:5, Funny)
I'm pretty sure Google's datacentre has evolved beyond the need for an ass.
Re: (Score:1)
I think they needed some to negotiate that slippery slope in China.
Re: (Score:2)
We have a couple PB in online storage just for our mainframe, much less online storage for Lotus Notes, a few thousand servers of varying OS's, speeds, and feeds, a large SAN that contains online backups for all of those servers, much less our tens of thousands of high density ta
Re: (Score:1)
come back when you have multi site workload balancing coupled with a full activity recovery plan !
Re: (Score:1)
Re: (Score:1)
Re:That's really not that large (Score:5, Funny)
A conversation overheard recently over the ether:
Lucas DC: Hi! I've got 11.38PB/s and 500TB!
Google DC: Hah! I've pulled bigger queries out of my back end.
...although I'm not quite sure what that says about Google's "interfacing preferences".
Meh....SDSC has 2 PetaBytes of online storage (Score:4, Informative)
Still....I like datacenters. The hum of equipment. 65 degree temps and lower. I once had my cube re-located to a tape library. Quiet...peaceful place
http://www.enterprisestorageforum.com/hardware/fe
Re: (Score:1)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
All the blinking lights, the spaghetti of cables. I love it.
I've actually never been in a datacenter. But I love to read articles like this one.
Hopefully one day I'll get a tour in one of these myself.
Re: (Score:1)
Velcro straps are a wonderful thing. They should be used liberally in the cabling of a cabinet, to avoid this spaghetti you speak of.
Oh, and touring a datacenter is interesti
Eh (Score:2)
Rhetorical bandwidth? (Score:5, Funny)
Another interpretation: (Score:2)
Re: (Score:1)
Or maybe it's what the Wifi-figures are. Rethorical Bandwith. "802.11g is 27mbps full duplex. That is 27mbps in each direction. So we have 27 rethorical mbps times two, which sums up to *drumroll* 54!
Re: (Score:2)
Hey, maybe we could have a new mod code, +0 Rhetorical. Making it so no-one can post a reply to it ^_^
Re: (Score:1)
300TB? (Score:1)
300TB of storage in a 10.000 square foot datacenter
Can fit 300TB in a single rack these days.. or is that a 10 square foot datacenter?
Re: (Score:2)
For example, assume you've got Sun Fire X4500 servers, and that you take them with the stock 500GB drives. 24TB per 4U server.
Now let's assume that you run RAIDZ2 on each server, dedicating 2 of the 48 drives to parity. That's 23TB per server. Now let's assume you want some redundancy, as in, completely separate failover capacity. You mirror every single server with hot standbys. Easy with ZFS, you can mirror the file system in real time without any issue
Re: (Score:2)
Now let's assume that you run RAIDZ2 on each server, dedicating 2 of the 48 drives to parity.
No-one would do this in a production environment where performance was even a passing concern (or if they did, they shouldn't have) - firstly, because parity-based RAID configurations are (relatively) slow and secondly because RAIDZ[2] (or parity-based in general, really) arrays shouldn't be any bigger than about 8 drives in total, or performance (and reliability) start to go downhill.
It's reasonable to assume yo
Re: (Score:2)
Let's not forget that the Sun X4500 is a quad-processor beast itself, and you're not going to saturate all 16 (possible) cores with storage demands, unless you're processing the data locally.
A more dedicated processor such as the Cell (which would seem to be ideally suited for software rendering) would red
Submitter (Score:5, Insightful)
Well passthecrackpipe, if you and your vast knowledge of large scale datacenters are not impressed with the story, why the hell did you submit it?
Re: (Score:2, Interesting)
Re: (Score:2)
Here is Nothing Interesting
This place you never heard of before is so incredibly irrelevant, it's almost surprising. Their moderate hype is somewhat misleading; if I hadn't mentioned it you might have been fooled, had you cared.
10.000 square foot datacenter is SMALL (Score:3, Informative)
Re: (Score:2)
Indeed, I have a larger closet in my apartment!
Re: (Score:1)
Unless they do mean 300TB in 10 square feet, that would be impressive!
Now if they can just get it down under $20K and under 2kW, then that's exactly what I need at home.
Can we find the drive Jar-Jar is on (Score:5, Funny)
OP doesn't seem very impressed... (Score:4, Interesting)
Re: (Score:1)
Actually, I think these systems generate the 'Jabba the Hutt' like features of Lucas we see on interviews. Those chin ripples look so real.
RHETORICAL? (Score:1, Interesting)
Penis.... er.... Data Envy? (Score:2, Insightful)
Re: (Score:2)
http://www.landoverbaptist.org/news0899/jar.html [landoverbaptist.org]
And they drive to work at 2400mph (Score:3, Insightful)
This means
A) they can push their entire storage through the network in 300*8Tb/(11Pb/s)=200ms.
or
B) the article author does not have a clue.
I think an anlogy would be: I drive back and forth to work everyday, or 400 times a year. My speed on each trip is 60mph, so in a year my speed is 60x400 or 24000mph.
Re: (Score:1)
Re: (Score:1)
I think the better analogy is: "You drive to work each day at light speed, despite work being less than an hour away." In other words, their entire data array can do the Kessel Run in less than 12 parsecs. Which is good if you are creating, sharing and batch rendering massive 3D and/or compositing fx files across a network.
Re: (Score:2)
data center (Score:1, Interesting)
which one makes the story... i think its down. (Score:2)
11.38 petabits? (Score:4, Interesting)
Of course, it could just be a coincidence.
Hardly a coincidence (Score:2, Informative)
Re: (Score:1)
Re: (Score:1)
For all the knocks of this center (Score:3, Insightful)
What I have found funny is the number of ppl who are speaking of how big their centers. Offhand, I tend to suspect that those centers could go on a MAJOR f%^&ing diet and need to have their budgets cut to a fifth. And finally, it is time to fire a bunch of the incompetents who can not run a tight center.
Re: (Score:1)
How can you state that other companies datacenters are too big and extremely wasteful when you have no
Re: (Score:1)
Speaking as just a 'regular folks' person, that sort of organization sounds strikingly like a big "athletic supporter" for a few big-balls management types. As such, it needs to have the fuck crushed out of it.
Nothing compared to my Sempron rig... (Score:3, Funny)
Why? Because my rig has never so much as contained - much less rendered - an image of Jar Jar Binks.
Pwned.
300TB in 10,000 sqft is a lot? (Score:3, Interesting)
Save the cheerleader, use the Force... (Score:2)
But? (Score:1, Funny)
Re: (Score:1)
60TB a movie...300TB total? (Score:1)
Re: (Score:1)
Re: (Score:2)
Re: (Score:1)
If they DID have the storage needed to have ALL their films onsite, and there was a 'net or physical security breach, wouldn't their loss/exposure be a lot larger?
Just a couple o' pennies worth...
Re: (Score:2)
If it's old, and stolen copies are already widely available, the losses probably aren't perceived as so immediate and crushing. That's not to say those losses aren't harmful in the long run--just that they aren't necessari
And it's in a national park (Score:2, Interesting)
There's considerable unhappiness in San Francisco about Lucasfilm's operation. It's in the Presidio, which used to be a military base and is now a national park. It's the only national park which has to make a profit, due to a Bush Administration deal. Letterman Army Hospital was torn down to make room for the Lucasfilm facility. The San Francisco Bay Guardian complains about this constantly, as they try to keep the Presidio from turning into an industrial park. The Lucasfilm move to the Presidio w
Pelosi's Presidio Re:And it's in a national park (Score:2)
Though anything in the SF Bay Guardian should be taken with a grain of salt, it should be noted that publication blames now-Speaker Nancy Pelosi [sfbg.com] (D-San Francisco) for the Presidio arrangement, not the Bush-41 Administration. Since the legislation was passed during the Clinton-42 administration, blaming it on either Bush is farfetched.
But the course taken wasn't unreasonable. The Presidio was already developed when it was a military base. Turning it into a traditional, naturalist national park would have r
Lucasfilm pay is mediocre too (Score:3, Interesting)
I suppose they were counting on the "cool factor". The job was cool, but not so cool I was willing to stick a stake through the heart of my family. Right after this, I read that Lucas donates 170 million to his alma mater. Hey George, why not donate 10% less and actually pay your people something more since you're insisting on setting up shop right in the freaking Presidio?
600 Tbyte of disk in total can't be right. I wrote an application a couple years ago that has 6 terabytes of disk allocated to it to cache its work. This was for a single app. Admittedly, we worked with fairly big data files where I was working, but I've got to think Lucasfilm's files are way larger than my 1-2 gig files.
Negotiation is important. (Score:2)
Re: (Score:2)
That goes for artists, too. Ever since Star Wars, ILM and Lucasfilm have inspired a lot of people to get into the industry. The first place the apply to is ILM. My guess is they've got so many applicants just yearning to do something on a high-profile movie that they can get away with low wages. (or at least 'low' relative to the cost of living there...) I honestly don't know the people there actually manage to work there and stay afloat. I've heard
Re: (Score:1)
Re: (Score:2)
Yes and no. From what I understand, they have their work flow set up sort of like an assembly line. A bunch of people work on very small chunks of the project. It's boiled down to a point that not a lot of 'talent' is needed so much as a pair of hands to work the mouse.
That's probably a gross over-simplification, but to me that sorta makes sense now. The work load is distributed to a LOT of people and there's more predictability and l
Prequels Scripts (Score:2)
He could have invested that same 10% in the world's best script writer for the Prequels and thereby realized a 10x ROI from now-former-fanboys actually buying the DVD's of his movies and thereby able to both raise pay and donate more money. But ego is a terrible vice.
This from a guy
Graphics processing power (Score:1)
http://en.wikipedia.org/wiki/UHDV [wikipedia.org]
http://en.wikipedia.org/wiki/22.2 [wikipedia.org]
Re: (Score:1)
Actually, it's impressive. Most impressive... (Score:4, Interesting)
It's a good sized datacenter, but what it's able to support in processing ability is the impressive part, and that the fat bandwidth runs at capacity almost all of the time by the demands of processing jobs. Proprietary software doles out jobs 24/7 to thousands of procs all over campus-- including artists' desktop machines-- for heavy duty computation: rendering and simulation and whatever it takes.
I can't imagine a facility where so many people are creating and pumping so much data around.
Their datacenter has a droid! (Score:4, Interesting)
The facility was absolutely beautiful. When going between two buildings on an overhead walkway I saw the Golden Gate bridge with a nice orange sunset behind it. I wish I had my camera with me.
They said that they have many dedicated OC-48 pipes to various studios and can handle just about any format, since every studio uses their own format. They convert it to their own internal format, which I believe they open sourced.
When they moved from Skywalker Ranch, it was completely seamless. They had an OC-192 (10gbps) link running between the old and new facility as more and more equipment was migrated to the new facility but people continued to work at the old one.
-Aaron
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
What's the point of building a data center with a beautiful scenic view? Computers can't see, and even if they could they wouldn't appreciate it.
Relative perspective (Score:1)
Hey, it's Lucasfilm, so it's automatically cool! (Score:2)
Re: (Score:1)
Theoreetical carrying capacity (Score:1)
Bullshit or Calculation Error : 569 NICs/server ! (Score:2, Informative)
11.38 Pbps is 11380 Tbps or 11380000 Gbps. This means that each
server has 569 network interfaces !! This is total bullshit. If
they had said they had 10*2000*2 = 40 Tbps, it would have been
based on more real (though irrelevant) data.
I hate it when ignorant journalists post meaningless data for public
consumption.
Willy
what a bunch of BS (Score:2)
To borrow from Fark.com (Score:2)
We want our nerd porn [urbandictionary.com]!
Re: (Score:1)
Where'd you come up with this resolution? I've never worked on a movie where the final rez was higher than 2K. I can only think of one set of elements I rendered at 4K -- a bunch of badly aliasing Mental Ray renders. A halfway decent renderer will let you get away with rendering at a lower rez than needed for the final comp.
Not to say your point isn't a good one though. Google's velcro and duct tape solution to server f
Re: (Score:2)
Just an idea.
Re: (Score:1)
And what the hell
Re: (Score:1)
Someone mod this troll down, please.