Inside the Lucasfilm datacenter

passthecrackpipe writes "Where can you find a (rhetorical) 11.38 petabits per second bandwidth? It appears to be inside the Lucasfilm Datacenter. At least, that is the headline figure mentioned in this report on a tour of the datacenter. The story is a bit light on the down-and-dirty details, but mentions a 10 gig ethernet backbone (adding up the bandwidth of a load of network connections seems to be how they derived the 11.38 petabits p/s figure. In that case, I have a 45 gig network at home.) Power utilization is a key differentiator when buying hardware, a "legacy" cycle of a couple of months, and 300TB of storage in a 10.000 square foot datacenter. To me, the story comes across as somewhat hyped up — "look at us, we have a large datacenter" kind of thing, "look how cool we are". Over the last couple of years, I have been in many datacenters, for banks, pharma and large enterprise to name a few, that have somewhat larger and more complex setups."

That's really not that large

[192939495969798999]

There are many corporate data centers larger and more powerful than that, it is much more impressive if the entire thing can run one giant application. Still, I'm pretty sure that Google's new datacenter wipes its ass with a datacenter the size of this one.

Re:Hmm?

[Anonymous Coward]

> everything, even every single rendered, uncompressed frame is stored on an Oracle database

WTF? Jabba the blob is not impressed, why would they do that? Wouldn't it make more sense to store metadata in the db and the actual image data on XFS RAID?

OP doesn't seem very impressed...

[been42]

So why submit this if you don't like it? Why not at least title it "Lucasfilm thinks it's soooo great."? I'm sure you've seen bigger data centers, and you can type 500 lines of code a minute, and maybe you defeated a ninja in hand-to-hand combat, but for the rest of us "normal" nerds it's still neat to read about the machines that get the work done in a business. Of course it's hyped up, it's a press release disguised as news. Take it for what it is, relax, and try to imagine those 2,000 servers in a secret cave under your house, manipulating the stock market in your favor. That's what I do.

RHETORICAL?

[Anonymous Coward]

How about theoretical? *yawn*

Re:Submitter

[Anonymous Coward]

Nomen est Omen.

data center

[ralph1]

Guess they have not been to a hospital data center yet. Should check out someone like dow chemical.

11.38 petabits?

[Nighttime]

As in reference to THX 1138 [imdb.com]?

Of course, it could just be a coincidence.

300TB in 10,000 sqft is a lot?

[willith]

The datacenter at one of my employer's satellite sites has four CLARiiONs, at 2 racks each, a 5-bay DMX-3, and a 4-bay XP1024, for 380TB raw, in 3,200 sqft, along with thirty racks of servers, a P595 mainframe, and several multi-rack computing clusters. There's plenty of cooling and it's really not THAT crowded. Managing to pack 10-12 racks of storage into a 10,000 sqft data center is not anything noteworthy.

And it's in a national park

[Animats]

There's considerable unhappiness in San Francisco about Lucasfilm's operation. It's in the Presidio, which used to be a military base and is now a national park. It's the only national park which has to make a profit, due to a Bush Administration deal. Letterman Army Hospital was torn down to make room for the Lucasfilm facility. The San Francisco Bay Guardian complains about this constantly, as they try to keep the Presidio from turning into an industrial park. The Lucasfilm move to the Presidio was something of a dot-com boom excess, when people thought SF was the place to be.

Pixar, in Emeryville, Tippett, in Berkeley, and Dreamworks, in Redwood City, are the innovative animation companies in the Bay Area. And of course, there's EA, SCEA, and some other game companies. Lucasfilm doesn't seem to get much attention.

There are data centers in San Francisco proper with far more storage, too. The Internet Archive has several petabytes of storage. There's a large colocation facility at the 6th St. offramp from I-280.

Lucasfilm pay is mediocre too

[rk]

They wanted me to move across the continent from a place with average cost of living and a 10 minute commute to work in San Francisco (right in the city, not even an outlying area) for about a 15% increase in pay. The only way I could afford that would be to take on a 2-3 hour commute and even then I'd have to run an even tighter ship, financially speaking, than I do now.

I suppose they were counting on the "cool factor". The job was cool, but not so cool I was willing to stick a stake through the heart of my family. Right after this, I read that Lucas donates 170 million to his alma mater. Hey George, why not donate 10% less and actually pay your people something more since you're insisting on setting up shop right in the freaking Presidio?

600 Tbyte of disk in total can't be right. I wrote an application a couple years ago that has 6 terabytes of disk allocated to it to cache its work. This was for a single app. Admittedly, we worked with fairly big data files where I was working, but I've got to think Lucasfilm's files are way larger than my 1-2 gig files.

Actually, it's impressive. Most impressive...

[Boss Sauce]

As somebody who (ab)uses that particular rig daily, the article misses the point about what's so awesome about the system.

It's a good sized datacenter, but what it's able to support in processing ability is the impressive part, and that the fat bandwidth runs at capacity almost all of the time by the demands of processing jobs. Proprietary software doles out jobs 24/7 to thousands of procs all over campus-- including artists' desktop machines-- for heavy duty computation: rendering and simulation and whatever it takes.

I can't imagine a facility where so many people are creating and pumping so much data around.

Their datacenter has a droid!

[AaronW]

I toured their new facility in San Francisco. They have over 300 10Gbps ports and all PCs are connected via gigabit. Their datacenter was 2/3 full of dual-Opteron servers running SuSE Linux (though they were considering switching). Their server room was spotless. No cables were visible anywhere, but I did see a Roomba moving about the floor. The fellow who ran it said that since they're ILM, they have to have droids.

The facility was absolutely beautiful. When going between two buildings on an overhead walkway I saw the Golden Gate bridge with a nice orange sunset behind it. I wish I had my camera with me.

They said that they have many dedicated OC-48 pipes to various studios and can handle just about any format, since every studio uses their own format. They convert it to their own internal format, which I believe they open sourced.

When they moved from Skywalker Ranch, it was completely seamless. They had an OC-192 (10gbps) link running between the old and new facility as more and more equipment was migrated to the new facility but people continued to work at the old one.

-Aaron