Forgot your password?
typodupeerror
IT

Inside the Lucasfilm datacenter 137

Posted by CmdrTaco
from the wrecking-childhoods-takes-bandwidth dept.
passthecrackpipe writes "Where can you find a (rhetorical) 11.38 petabits per second bandwidth? It appears to be inside the Lucasfilm Datacenter. At least, that is the headline figure mentioned in this report on a tour of the datacenter. The story is a bit light on the down-and-dirty details, but mentions a 10 gig ethernet backbone (adding up the bandwidth of a load of network connections seems to be how they derived the 11.38 petabits p/s figure. In that case, I have a 45 gig network at home.) Power utilization is a key differentiator when buying hardware, a "legacy" cycle of a couple of months, and 300TB of storage in a 10.000 square foot datacenter. To me, the story comes across as somewhat hyped up — "look at us, we have a large datacenter" kind of thing, "look how cool we are". Over the last couple of years, I have been in many datacenters, for banks, pharma and large enterprise to name a few, that have somewhat larger and more complex setups."
This discussion has been archived. No new comments can be posted.

Inside the Lucasfilm datacenter

Comments Filter:
  • Rendering (Score:2, Funny)

    by Anonymous Coward
    Only a few boxen are used rendering and effects. The rest is to track and calculate sales of Star Wars merchandise.
  • 300tb (Score:1, Insightful)

    by Anonymous Coward
    Is that all? Most datacenters that house more than 1 large customer usually starts at about 300tb, nothing to write home about. Most customers using sap use a lot more.
    • Is that all? Most datacenters that house more than 1 large customer usually starts at about 300tb, nothing to write home about.
      Yeah, I was a bit disappointed as well.

      By mid-year, my pre-production lab will have 150TB. Our production datacenter, just for PLM alone, has something like half a petabyte.

  • passthecrackpipe writes:

    Over the last couple of years, I have been in many datacenters, for banks, pharma and large enterprise to name a few, that have somewhat larger and more complex setups."
  • I'll just assume it runs linux. Did it say in TFA?
    • by hjf (703092)
      On Oracle Magazine ( http://www.oracle.com/technology/oramag/oracle/06- may [oracle.com] ), they said they used Linux for their "render farm" (I hate the word farm, computers aren't cattle), but designs were made in other platforms (SGI, Mac...). Finally, everything, even every single rendered, uncompressed frame is stored on an Oracle database (which runs on Linux).
      • Re: (Score:1, Interesting)

        by Anonymous Coward
        > everything, even every single rendered, uncompressed frame is stored on an Oracle database

        WTF? Jabba the blob is not impressed, why would they do that? Wouldn't it make more sense to store metadata in the db and the actual image data on XFS RAID?
        • by hjf (703092)
          don't ask me. They claim to have a 300TB Oracle database. whatever works for them, right? anyway check the article I posted, maybe I misunderstood it.
      • by elfurbe (759480)
        "I hate the word farm, computers aren't cattle"

        If they were cattle, it'd be a render ranch. Render farm implies a soil metaphor to me. Every machine I add to my render farm is arable soil, waiting to be planted with precious digital seed to yield my crop of rendered data. A ranching metaphor is much harder to construct.
        • by hjf (703092)
          ok. first: what the fuck? second: "A farm is the basic unit in agriculture. It is a section of land devoted to the production and management of food, either produce or livestock." (from wikipedia).
    • Re:Hmm? (Score:5, Funny)

      by gEvil (beta) (945888) on Sunday January 28, 2007 @10:38AM (#17789612)
      Nope. They run LucasOS. It's perfect for their needs, since it's constantly being updated to suit his vision.
    • It runs SCUMM, of course.
  • by 192939495969798999 (58312) <.info. .at. .devinmoore.com.> on Sunday January 28, 2007 @10:22AM (#17789552) Homepage Journal
    There are many corporate data centers larger and more powerful than that, it is much more impressive if the entire thing can run one giant application. Still, I'm pretty sure that Google's new datacenter wipes its ass with a datacenter the size of this one.
    • by rtaylor (70602) on Sunday January 28, 2007 @10:27AM (#17789574) Homepage
      Still, I'm pretty sure that Google's new datacenter wipes its ass with a datacenter the size of this one.

      I'm pretty sure Google's datacentre has evolved beyond the need for an ass.
      • I'm pretty sure Google's datacentre has evolved beyond the need for an ass.

        I think they needed some to negotiate that slippery slope in China.

    • by Zen (8377)
      Yeah - that datacenter is nothing. I don't consider ours that big either, but the company I work for (non profit in the healthcare insurance industry) would be ranked around #40 on the global fortune 500 list if we were for profit.

      We have a couple PB in online storage just for our mainframe, much less online storage for Lotus Notes, a few thousand servers of varying OS's, speeds, and feeds, a large SAN that contains online backups for all of those servers, much less our tens of thousands of high density ta
      • I know ours would blow this away without even blinking, and we're out of space and have already broken ground to double our size.
        supersized as it is, one datacenter is nothing when it's alone :)

        come back when you have multi site workload balancing coupled with a full activity recovery plan !

        • by Zen (8377)
          Yeah - that is already being worked on. That is the current top priority for the senior executive management. We expect it to be finished by late 2008.
    • by sgt_doom (655561)
      I'd have to agree. Ever check out Organized Crime's data centers? We're talking super hugh, here.....
    • by geobeck (924637) on Sunday January 28, 2007 @02:49PM (#17791072) Homepage

      ...I'm pretty sure that Google's new datacenter wipes its ass with a datacenter the size of this one.

      A conversation overheard recently over the ether:

      Lucas DC: Hi! I've got 11.38PB/s and 500TB!

      Google DC: Hah! I've pulled bigger queries out of my back end.

      ...although I'm not quite sure what that says about Google's "interfacing preferences".

  • by Danathar (267989) on Sunday January 28, 2007 @10:24AM (#17789560) Journal
    San Diego Super Computing Center (SDSC) has 2 Petabytes of online Storage with 400TB for researchers. They have 18PB of archival tape storage.

    Still....I like datacenters. The hum of equipment. 65 degree temps and lower. I once had my cube re-located to a tape library. Quiet...peaceful place

    http://www.enterprisestorageforum.com/hardware/fea tures/print.php/3634881 [enterprise...eforum.com]
    • by wik (10258)
      Hum of equipment? Either it's not much of a datacenter or your hearing is already shot.
    • by fons (190526)
      I hear you.

      All the blinking lights, the spaghetti of cables. I love it.

      I've actually never been in a datacenter. But I love to read articles like this one.

      Hopefully one day I'll get a tour in one of these myself.

      • Please, if you love the spaghetti of cables, NEVER go anywhere NEAR a datacenter I have to work in. Those who believe in spaghetti cabling should be strung up in it and left to die slowly, suspended above the air vents in the cold aisle so they dry out and preserve nicely as a warning to others who might be tempted to spaghetti-cable.

        Velcro straps are a wonderful thing. They should be used liberally in the cabling of a cabinet, to avoid this spaghetti you speak of.

        Oh, and touring a datacenter is interesti
  • by Peter Cooper (660482) * on Sunday January 28, 2007 @10:25AM (#17789564) Homepage Journal
    Is that the speed you can talk at?
    • A rhetorical question does not expect an answer...

      ...so maybe "rhetorical bandwidth" is a nice way of saying that the data flows only in one direction? ;-)
      • by ozamosi (615254)
        Like half duplex or something?

        Or maybe it's what the Wifi-figures are. Rethorical Bandwith. "802.11g is 27mbps full duplex. That is 27mbps in each direction. So we have 27 rethorical mbps times two, which sums up to *drumroll* 54! :D"
      • by Barny (103770)
        Nope, its just UDP ;)

        Hey, maybe we could have a new mod code, +0 Rhetorical. Making it so no-one can post a reply to it ^_^
      • by mrscorpio (265337)
        Yeah, I think he means "theoretical" but just wanted to sound smart.
  • 300TB of storage in a 10.000 square foot datacenter

    Can fit 300TB in a single rack these days.. or is that a 10 square foot datacenter?

  • Submitter (Score:5, Insightful)

    by kevin_conaway (585204) on Sunday January 28, 2007 @10:41AM (#17789620) Homepage

    Well passthecrackpipe, if you and your vast knowledge of large scale datacenters are not impressed with the story, why the hell did you submit it?

    • Re: (Score:2, Interesting)

      by Anonymous Coward
      Nomen est Omen.
    • by Vreejack (68778)
      Let's play editor and re-word this summary

      Here is Nothing Interesting

      This place you never heard of before is so incredibly irrelevant, it's almost surprising. Their moderate hype is somewhat misleading; if I hadn't mentioned it you might have been fooled, had you cared.

  • by Secrity (742221) on Sunday January 28, 2007 @10:58AM (#17789702)
    10.000 square feet for a datacenter is not very impressive. The datacenter that I work in did a relatively modest 100,000 square foot EXPANSION which was the result of absorbing an adjoining atrium. I suspect that the power equipment and air handlers may take up 10,000 square feet.
    • 10.000 square feet for a datacenter is not very impressive.

      Indeed, I have a larger closet in my apartment!

  • by $RANDOMLUSER (804576) on Sunday January 28, 2007 @11:01AM (#17789724)
    and format it?
  • by been42 (160065) on Sunday January 28, 2007 @11:10AM (#17789778) Homepage
    So why submit this if you don't like it? Why not at least title it "Lucasfilm thinks it's soooo great."? I'm sure you've seen bigger data centers, and you can type 500 lines of code a minute, and maybe you defeated a ninja in hand-to-hand combat, but for the rest of us "normal" nerds it's still neat to read about the machines that get the work done in a business. Of course it's hyped up, it's a press release disguised as news. Take it for what it is, relax, and try to imagine those 2,000 servers in a secret cave under your house, manipulating the stock market in your favor. That's what I do.
    • "Take it for what it is, relax, and try to imagine those 2,000 servers in a secret cave under your house, manipulating the stock market in your favor. That's what I do."

      Actually, I think these systems generate the 'Jabba the Hutt' like features of Lucas we see on interviews. Those chin ripples look so real.
  • RHETORICAL? (Score:1, Interesting)

    by Anonymous Coward
    How about theoretical? *yawn*
  • Why all the negativity toward Lucas? Jar Jar's dead man, let it go. George said he was sorry already. I think it's a good story. It's absolutely fascinating to me to see how they make movies today, how much data gets pushed around, and how they make sure that the creative people have access to what they need, when they need it. And they do all this to support incredible time schedules, with boatloads of cash riding on every second. I don't know how anyone can say that this isn't an impressive operation. A
  • by viking80 (697716) on Sunday January 28, 2007 @11:53AM (#17790048) Journal
    300TB storage and 11 petabits/s bandwidth.

    This means

    A) they can push their entire storage through the network in 300*8Tb/(11Pb/s)=200ms.
    or
    B) the article author does not have a clue.

    I think an anlogy would be: I drive back and forth to work everyday, or 400 times a year. My speed on each trip is 60mph, so in a year my speed is 60x400 or 24000mph.

    • It's more like "There are 10,000 cars in this city, driving at 30 mph, so there speed is 300,000 mph!"...
    • > I think an anlogy would be: I drive back and forth to work everyday, or 400 times a year. My speed on each trip is 60mph, so in a year my speed is 60x400 or 24000mph.

      I think the better analogy is: "You drive to work each day at light speed, despite work being less than an hour away." In other words, their entire data array can do the Kessel Run in less than 12 parsecs. Which is good if you are creating, sharing and batch rendering massive 3D and/or compositing fx files across a network.
    • by Guspaz (556486)
      More like, in New York City, 10 million people commute to work each day (making up figures here), each traveling down the road at 50km/h. So the aggregate commute speed in New York City is 46% the speed of light!
  • data center (Score:1, Interesting)

    by ralph1 (900228)
    Guess they have not been to a hospital data center yet. Should check out someone like dow chemical.
  • 11.38 petabits? (Score:4, Interesting)

    by Nighttime (231023) on Sunday January 28, 2007 @12:15PM (#17790156) Homepage Journal
    As in reference to THX 1138 [imdb.com]?

    Of course, it could just be a coincidence.
  • by WindBourne (631190) on Sunday January 28, 2007 @12:20PM (#17790190) Journal
    I have to wonder how many systems they have? They accomplish a great deal with what is a fairly small area. I would guess that they each computer has major ram and is simply NFSed back to a central server.

    What I have found funny is the number of ppl who are speaking of how big their centers. Offhand, I tend to suspect that those centers could go on a MAJOR f%^&ing diet and need to have their budgets cut to a fifth. And finally, it is time to fire a bunch of the incompetents who can not run a tight center.
    • by Zen (8377)
      I'm not following you here. Yes, I am one of those who responded with some rough stats about the datacenter I work at. I also stated that I didn't even think mine was that great (big was the word I used). Because it's not. But it beats the crap out of the Lucas one, which is the story, so when you can relate to it and build on the topic, then it is an ontopic post and adds to the topic of conversation.

      How can you state that other companies datacenters are too big and extremely wasteful when you have no
      • When a single hour of downtime in your datacenter costs your company over $4,000,000 not to mention loss of brand name status, competitive edge in mergers and acquistions, and other non-tangible costs, you tend not to sweat the 'small stuff' - like a $100M datacenter.

        Speaking as just a 'regular folks' person, that sort of organization sounds strikingly like a big "athletic supporter" for a few big-balls management types. As such, it needs to have the fuck crushed out of it.

        A large company who dramatically

  • by gatkinso (15975) on Sunday January 28, 2007 @12:26PM (#17790230)
    ...running FC6 x64.

    Why? Because my rig has never so much as contained - much less rendered - an image of Jar Jar Binks.

    Pwned.

  • by willith (218835) * on Sunday January 28, 2007 @12:35PM (#17790262) Homepage
    The datacenter at one of my employer's satellite sites has four CLARiiONs, at 2 racks each, a 5-bay DMX-3, and a 4-bay XP1024, for 380TB raw, in 3,200 sqft, along with thirty racks of servers, a P595 mainframe, and several multi-rack computing clusters. There's plenty of cooling and it's really not THAT crowded. Managing to pack 10-12 racks of storage into a 10,000 sqft data center is not anything noteworthy.
  • Now I'm disappointed. I had hoped Masi Oka [wikipedia.org] would be working there.
  • But? (Score:1, Funny)

    by Anonymous Coward
    Will it run Vista? Sounds like they might need to upgrade!
    • There is yet another University surplus equipment auction coming up the end of March that I am looking forward to. Because I have enjoy messing around with older equipment. I am hoping that 'Vista' will push more good boxes my way at low-low prices. Not that aging x86 boxes is my main interest (unless it's REALLY old and runs CP/M-86 or Microsoft Xenix.) One of the things I specialize at the auction in is the weird old Unix stuff. Sun and SGI boxes that none of the 'jelly stain on the necktie' noneck b
  • Does anybody else find it questionable that he said Pirates of the Caribbean required approx. 50TB of storage, and the next one will require 25% more....but then goes on to say that there is total storage space of 300TB in the data center. Thats basically enough to store six movies of equivalent size to Pirates, so where are all the rest of the movies they make stored??
    • You mean the rest of the movies they have ever made or the rest of the movies they are making at the same time? As soon as a movie is done all of the data is offlined to backup storage. 300TB is for the 2-6 movies they tend to be working on at a time.
    • by VENONA (902751)
      Above, an AC posted a link to http://www.oracle.com/technology/oramag/oracle/06 - may/o36lucas.html [oracle.com] which sez the answer to your question is 'tape'. Makes sense, I suppose. Storing old movies which require TB don't sound like something to store on- or near-line. I doubt much of it is reusable, on a day to day basis. When you launch a major project (make Greedo shoot first or something) for it, then it's in the books, and you have a business requirement to fill on-line storage, acquire more if you need it, et
      • A thought I just had...

        If they DID have the storage needed to have ALL their films onsite, and there was a 'net or physical security breach, wouldn't their loss/exposure be a lot larger?

        Just a couple o' pennies worth...
        • by VENONA (902751)
          Could be, but I doubt it. In the case of media stuff, the loss is probably proportional to immediacy. If a game, movie, CD, even an operating system (Google on Vista leaks) leaks widely, just as the thing is supposed to be ramping sales in huge way, it has to hurt, say, stock share prices.

          If it's old, and stolen copies are already widely available, the losses probably aren't perceived as so immediate and crushing. That's not to say those losses aren't harmful in the long run--just that they aren't necessari
  • There's considerable unhappiness in San Francisco about Lucasfilm's operation. It's in the Presidio, which used to be a military base and is now a national park. It's the only national park which has to make a profit, due to a Bush Administration deal. Letterman Army Hospital was torn down to make room for the Lucasfilm facility. The San Francisco Bay Guardian complains about this constantly, as they try to keep the Presidio from turning into an industrial park. The Lucasfilm move to the Presidio w

    • Though anything in the SF Bay Guardian should be taken with a grain of salt, it should be noted that publication blames now-Speaker Nancy Pelosi [sfbg.com] (D-San Francisco) for the Presidio arrangement, not the Bush-41 Administration. Since the legislation was passed during the Clinton-42 administration, blaming it on either Bush is farfetched.

      But the course taken wasn't unreasonable. The Presidio was already developed when it was a military base. Turning it into a traditional, naturalist national park would have r

  • by rk (6314) * on Sunday January 28, 2007 @01:54PM (#17790686) Journal
    They wanted me to move across the continent from a place with average cost of living and a 10 minute commute to work in San Francisco (right in the city, not even an outlying area) for about a 15% increase in pay. The only way I could afford that would be to take on a 2-3 hour commute and even then I'd have to run an even tighter ship, financially speaking, than I do now.

    I suppose they were counting on the "cool factor". The job was cool, but not so cool I was willing to stick a stake through the heart of my family. Right after this, I read that Lucas donates 170 million to his alma mater. Hey George, why not donate 10% less and actually pay your people something more since you're insisting on setting up shop right in the freaking Presidio?

    600 Tbyte of disk in total can't be right. I wrote an application a couple years ago that has 6 terabytes of disk allocated to it to cache its work. This was for a single app. Admittedly, we worked with fairly big data files where I was working, but I've got to think Lucasfilm's files are way larger than my 1-2 gig files.
    • We techies really sux at Negotiations. Sadly, the more hard core you are, the less business savey we appear to be. I have been stuck around 100K. A friend of mine with less education and experience was offered a job at MS. He was originally offered 85K (this was 8 years ago). He said no and held out for 150K, stock options, and benefits. They came around and re-offered him. I do not know exactly what it was (per contract, he was not allowed to say), but he says that it was more than what he wanted. After se
    • "I suppose they were counting on the "cool factor"."

      That goes for artists, too. Ever since Star Wars, ILM and Lucasfilm have inspired a lot of people to get into the industry. The first place the apply to is ILM. My guess is they've got so many applicants just yearning to do something on a high-profile movie that they can get away with low wages. (or at least 'low' relative to the cost of living there...) I honestly don't know the people there actually manage to work there and stay afloat. I've heard
      • by GnuDiff (705847)
        Apparently there is much more talent around than previously thought?
        • "Apparently there is much more talent around than previously thought?"

          Yes and no. From what I understand, they have their work flow set up sort of like an assembly line. A bunch of people work on very small chunks of the project. It's boiled down to a point that not a lot of 'talent' is needed so much as a pair of hands to work the mouse.

          That's probably a gross over-simplification, but to me that sorta makes sense now. The work load is distributed to a LOT of people and there's more predictability and l
    • Right after this, I read that Lucas donates 170 million to his alma mater. Hey George, why not donate 10% less and actually pay your people something more since you're insisting on setting up shop right in the freaking Presidio?

      He could have invested that same 10% in the world's best script writer for the Prequels and thereby realized a 10x ROI from now-former-fanboys actually buying the DVD's of his movies and thereby able to both raise pay and donate more money. But ego is a terrible vice.

      This from a guy
  • The majority of Lucasfilm's processing power is used for Graphics generation for ILM (unlike say Google). I think the hidden message of this article is they could get a huge screen and projector, play, for example, Crysis on full settings including 64x Antialiasing and Anisotropic Filtering, at 4320p with 22.2 surround sound and say to Sony, "Thats TrueHD!"
    http://en.wikipedia.org/wiki/UHDV [wikipedia.org]
    http://en.wikipedia.org/wiki/22.2 [wikipedia.org]
  • by Boss Sauce (655550) on Sunday January 28, 2007 @02:32PM (#17790960) Homepage Journal
    As somebody who (ab)uses that particular rig daily, the article misses the point about what's so awesome about the system.

    It's a good sized datacenter, but what it's able to support in processing ability is the impressive part, and that the fat bandwidth runs at capacity almost all of the time by the demands of processing jobs. Proprietary software doles out jobs 24/7 to thousands of procs all over campus-- including artists' desktop machines-- for heavy duty computation: rendering and simulation and whatever it takes.

    I can't imagine a facility where so many people are creating and pumping so much data around.
  • by AaronW (33736) on Sunday January 28, 2007 @02:34PM (#17790966) Homepage
    I toured their new facility in San Francisco. They have over 300 10Gbps ports and all PCs are connected via gigabit. Their datacenter was 2/3 full of dual-Opteron servers running SuSE Linux (though they were considering switching). Their server room was spotless. No cables were visible anywhere, but I did see a Roomba moving about the floor. The fellow who ran it said that since they're ILM, they have to have droids.

    The facility was absolutely beautiful. When going between two buildings on an overhead walkway I saw the Golden Gate bridge with a nice orange sunset behind it. I wish I had my camera with me.

    They said that they have many dedicated OC-48 pipes to various studios and can handle just about any format, since every studio uses their own format. They convert it to their own internal format, which I believe they open sourced.

    When they moved from Skywalker Ranch, it was completely seamless. They had an OC-192 (10gbps) link running between the old and new facility as more and more equipment was migrated to the new facility but people continued to work at the old one.

    -Aaron
    • by kv9 (697238)
      XS4ALL also has a droid [iwans.net]
    • by antdude (79039)
      Where was it? I missed it.
    • The facility was absolutely beautiful. When going between two buildings on an overhead walkway I saw the Golden Gate bridge with a nice orange sunset behind it.

      What's the point of building a data center with a beautiful scenic view? Computers can't see, and even if they could they wouldn't appreciate it.

  • The point of the story was to display ILM data crunching power as impressive for a POST PRODUCTION house. Not "the greatest data center in the world". Compared to any other post production house, ILM is pretty darn impressive.
  • Nevermind the fact that there are much larger and complex setups out there, as others have pointed out. Nevermind the fact that Star Wars was a ripoff of a Japanese pulp science fiction novel.
  • Theoretical bandwidth is a chimera. All the cars on Los Angelos freeways at a given time, carrying boxes of tapes -- now that's some theoretical bandwidth. What matters is achieved write and read capacity -- I believe the record [gridtoday.com] is 14.5 Gb/s sustained.
  • TFA talks about 2000 servers equipped with 10 Gbps network cards.
    11.38 Pbps is 11380 Tbps or 11380000 Gbps. This means that each
    server has 569 network interfaces !! This is total bullshit. If
    they had said they had 10*2000*2 = 40 Tbps, it would have been
    based on more real (though irrelevant) data.

    I hate it when ignorant journalists post meaningless data for public
    consumption.

    Willy
  • Imageworks has almost twice as much gear as they have. They are just blowing smoke out there ass.
  • * This thread is useless without pics! *

    We want our nerd porn [urbandictionary.com]!

I am the wandering glitch -- catch me if you can.

Working...