Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
Supercomputing IT

Weta Digital Grows Cluster 209

Posted by CmdrTaco from the thats-a-whole-lotta-muscle dept.
Korgan writes "A little over 3 years after their last upgrade, Weta Digital has just added another 250 more blade servers to their render farm to help with the final renderings of King Kong. From the article: "The IBM Xeon blade servers, each with two 3.4 gigahertz processors and 8 gigabytes of memory, are housed at the New Zealand Supercomputing Centre in central Wellington. They have been added to the centre's existing bank of 1144 Intel 2.8GHz processors, boosting its power by 50 per cent to create a supercomputer with the equivalent power of nearly 15,000 PCs. The servers run the Red Hat version of the open-source Linux operating system. The purchase means the centre is back among the 100 largest supercomputing clusters in the world." And all that computing power is still available for hire when Peter Jackson isn't using it."
This discussion has been archived. No new comments can be posted.

Weta Digital Grows Cluster More

Weta Digital Grows Cluster

Comments Filter:

  • Export restrictions? (Score:5, Interesting)

    by BWJones ( 18351 ) * on Tuesday October 11, 2005 @12:44PM (#13765944) Homepage Journal
    So, out of curiosity. What happened to the export restrictions of the US government on CPU's beyond a certain MIPS range? I remember that the old PowerMac 9600/300 eclipsed this federally mandated figure and now we have home game consoles that easily eclipse that performance range. Certainly the advent of cluster computing with commodity hardware made many of these issues moot, but what is the status of the law? Was it repealed or is it just commonly ignored?

    I know that historically, NeXT did quite a bit of work for TLA agencies and that Richard Crandall's program, zilla.app grabbed some attention from interested parties. Because of this work, NeXT had some cash infusion for their hardware even after shutting the line down for general commercial consumption. More recently, Apple has been selling Xserves to some of those same agencies, and contractors for work, but I do not know if they are selling any clusters outside the US?

    The history of course behind this law was that the CIA and NSA were concerned that foreign governments could use compute time to help design nuclear weapons as well as defeat cryptography that might compromise US secrets.

    • These laws may still be in place. But, businesses have evolved to be international companies. This could have been done by a non-us division. They designed, built, and sold it with no us division involved then US law would not apply. Or, am I totally off base here?
      • IANAL, but, as I understand it, with the shipping restrictions, they only apply with shipping those computers to certain companies. If you were shipping to, say, North Korea, Iran, or any number of potentially third-world dictatorships, then you'd probably have problems. However, if you're shipping to Austraila or New Zealand, you probably wouldn't have any problems.
      • Since when does US care about foreign nations laws?

    • Re:Export restrictions? (Score:5, Informative)

      by 14erCleaner ( 745600 ) <FourteenerCleaner@yahoo.com> on Tuesday October 11, 2005 @12:53PM (#13766016) Homepage Journal
      They've been eased over the years, e.g. here is a list of articles [justfuckinggoogleit.com].

      In 2002 it was upped to 195,000 million theoretical operations per second, and the limit goes up automatically every six months. A typical PC in 2002 was 2000 MTOPs, so this allows export of some rather big honking systems.

    • . . . export restrictions are generally overcome by the all mighty dollar, and besides hasn't New Zealand merged with Hollywood now?

      More interestingly, can anyone see digital actors quickly surpassing their organic cousins, no matter what Peter Jackson says [kongisking.net]?

      And slightly more interestingly, when will New Zealand surpass California in flim making, it is the ideal location, with better light, more interesting geography, and (at the moment) far cheaper to work in. There are of course the problems with the re
    • Intel have plants all over the world [cio-asia.com], I don't think any US export restrictions are going to hamper them selling chips to anywhere. I don't know whether US export restrictions are meant to apply to IP or the physical product, but the fact that they have plants in China suggest neither, as the Chinese aren't going to let the US stop them sending stuff to N Korea if they want to.
    • When very few of the parts are actually made in the US, why would US Export Restrictions apply?

  • Clearly... (Score:5, Funny)

    by TooMuchEspressoGuy ( 763203 ) on Tuesday October 11, 2005 @12:47PM (#13765962)
    Clearly we need a Beowulf cluster to slay this gigantic King Kong cluster!

  • Blah (Score:4, Funny)

    by Anonymous Coward on Tuesday October 11, 2005 @12:47PM (#13765963)
    That's nothing. The people arrested in the story earlier today had the computing power of 100,000 personal computers. Beat that!

    • Re:Blah (Score:3, Interesting)

      by sleeper0 ( 319432 )
      How do you combine 1644 server class ~3ghz CPUss and end up with the power of 15,000 PCs? Only in the marketing department...
      • That must PCs with x286 processors.

      • Extra-Beefy Power Supply perhaps??? (Score:3, Funny)

        by Anonymous Coward
        Power of 1 PC: 250 Watts
        Power of 15,000 PCs: Enough to power a small town

        Ability to do math in your head: Priceless

        For everything else, there's xcalc.
      • The 15,000 probably wasn't calculated by multiplying GHz -- it's probably done with processing time. RAM would play a huge role in video rendering operations.

        Can their 1.6k server machines really equal the computational time of 15k average PCs? I'm not sure, but I wouldn't doubt it.

  • power draw (Score:5, Funny)

    by dreadlocks ( 637491 ) on Tuesday October 11, 2005 @12:48PM (#13765982)
    that's why the lights dim in Wellington when the cluster is rendering

    • that's why the lights dim in Wellington when the cluster is rendering

      And there was me thinking it was the rats [slashdot.org] again!

    • Re:power draw (Score:4, Funny)

      by Farmer Tim ( 530755 ) on Tuesday October 11, 2005 @01:52PM (#13766518) Journal
      You have no idea how hard it is for an Australian to resist making a joke about dim lights in Wellington having nothing to do with the power grid...oh, whoops...
    • Funny, this is the first thing I thought of when I read about their choice of processors. They really picked the most inefficient chips before (P4 2.8G) and just did it again with P4-based Xeons. For this application, it seems to me that Opterons would be the no-brainer choice. Go figure!

    • Re:power draw (Score:4, Interesting)

      by shokk ( 187512 ) <ernieoporto AT yahoo DOT com> on Tuesday October 11, 2005 @02:41PM (#13766931) Homepage Journal
      Seriously, I would love to see an article, not on what their render farm is like, but on what their power management for it is like. Do they use remote-managed power strips or are they all just popped in, hoping that they don't blow a fuse. How many UPSs do they have and what kind of on-battery runtime do they shoot for?

      What kind of power draw do the blade chassis have? What blades? What version of Red Hat?!?!?!

      Unfortunately TFA is very short on details and reads more like "Peter Jackson went out and bought 500 computers! Woo!"

  • Explain this "new" math to me... (Score:5, Insightful)

    by pla ( 258480 ) on Tuesday October 11, 2005 @12:53PM (#13766026) Journal
    They have been added to the centre's existing bank of 1144 Intel 2.8GHz processors, boosting its power by 50 per cent to create a supercomputer with the equivalent power of nearly 15,000 PCs

    Total processors: 1644.

    Now, the Xeons do a bit better than the run-of-the-mill P4, but 10x faster? No way.

    For that matter, they don't run faster at all. They just do somewhat better (as in, 10-25%, not 913%) on certain types of memory-heavy tasks.

    Someone either made a major typo or pulled numbers from their netherregion...
    • Not surprising.

      You have 1144 2.4 GHz CPU's in the old cluster.
      You add 250 dual 3.4GHz Xeon blade systems.
      You also have an increase of 50% in performance.

      That means that 250 of the new systems is roughly equal to 572 of the old systems.

      If they are single CPU 2.8 GHz P4's that mean it's just 250*2 = 500 3.4 GHz CPUs to 572 2.4 GHz CPUs. Or a 14.4% increase in performance per CPU for the upgrade of 2.8 GHz to 3.4 GHz (a 21% increase in CPU clock speed).

    • Re:Explain this "new" math to me... (Score:5, Informative)

      by brunes69 ( 86786 ) <`gro.daetsriek' `ta' `todhsals'> on Tuesday October 11, 2005 @01:06PM (#13766150)
      The 250 added blade servers each have two 3.4 GHz CPus, while the existing 1144 only had two 2.6 GHz CPUs.

      Every two new servers is approx. as powerful as three old servers. It is more like they are now running 1519 dual 2.4 Ghz machines, or 3038 2.4 GHz cores.

      Also, remember that a 2.4 GHz is faster than two 1.2 Ghz chips, because of instruction set improvements.

      So, I would not say it would be far off to say that this cluster is approx. the computing power of 15,000 1 - 1.5 GHz machines. This is probably what they are basing the numbers off of.
      • Also, remember that a 2.4 GHz is faster than two 1.2 Ghz chips, because of instruction set improvements.

        Bzzzt. Better luck next time. A 2.4 ghz P4 is not faster than 2 1.2 ghz PIII's. The PIII had a higher IPC. Instruction set improvements? Instruction sets don't do much for performance either way; architecture does. Even if you're refererring to the evolving set of SIMD instructions you are still getting fewer things done on modern processors per clock cycle than with a generation or two back.
    • No, actually they're a gross underestimate, because you see, they're comparing to the power of 15,000 original IBM PC's.
    • And the more serious reply:

      You can get greater than linear speedup on a system like this because you can keep more of your task in memory. Imagine you need to render 1000G of frames. This system probably has 1000G of memory, whereas any desktop system would have to move that data onto and off of a disk, which is > an additional 10x slower. So hypothetically, it is possible that this monster system renders at 15k times the speed they can get off of a single pc.

      Of course, I believe they just erred in th
      • You can get greater than linear speedup on a system like this because you can keep more of your task in memory.

        This in supercomputing circles is called superlinear speedup and it is defined that by using X number of processors, you get performance greater than X*n processors. And, to my knowledge it is only achieved when applications are able to keep a good part of their program on the processors cache, not the main system memory. If memory was the limitation and not CPU power, then throwing a ton of memo
        • And, to my knowledge it is only achieved when applications are able to keep a good part of their program on the processors cache, not the main system memory.

          Exactly, which is why P4 Xeons still out perform the AMD Opteron for some memory intensive tasks - faster clock rate -> faster cache speed.

          The big problem with the P4 architecture is how often the CPU is starved for data, however, something that AMD Opterons (especially the dual core) are able to overcome.

          If memory was the limitation and not CP

        • But it gets quite expensive to throw memory AND processors into one box if you need both to make your speedup happen. Note that they're already putting a fair amount (8G) into these boxes. It's comparatively not cheap to buy a box that will go beyond 8G.
    • equivalent power of nearly 15,000 PCs ...

      Total processors: 1644

      Now, the Xeons do a bit better than the run-of-the-mill P4, but 10x faster? No way.

      I caught the math/editorial error as well. I'm guessing its supposed to be the equivalent to 1,500 PCs, that is what I would say. Also, AFAIK Xeons and Pentiums are the same besides the unlocking of the SMP mechanism and more options for cache which may help specific applications. A guy I work with has benchmarked Xeons and Pentiums, and has found that clock pe
    • The new stuff runs 64 bit code and also can address larger blocks of memory.

      The old stuff was limited to 32 bit code.

    • These are Xeons. I think they left out the word "consumption": ... eqivalent power consumption of nearly 15,000 PCs.
    • Just use the same "average computer" performance values you did 10 years (or 5 years) ago.
    • Now, the Xeons do a bit better than the run-of-the-mill P4, but 10x faster? No way.

      You have to understands this is rendering and not actual tasks of running a multi-threaded desktop environment. If they were using something like Maya3d or their own inhouse app... The answer is yes way.

      When you render to 3d it uses all of the cpu and every cpu you have and every register on the cpu and cache if the rendering software is up to snuff. So what you are looking for is raw computer horsepower. Each cpu can effecti
      • Actually Maya is only used for the interactive work. The batch 3D rendering Weta Digital uses is PRMan, and Shake for batch compositing.

        Also it doesn't matter if you are using all the cache in the CPUs, say for example if the shots you're rendering use texture maps of a 100 MBs and the RIB file might be a few hundred megabytes as well (so that also causes some bandwidth bottleneck). Of course you are can exploit some coherence in data (between frames or even in the same frame). So in the end you won't get t
  • No matter how many CPUs they throw at it, Jackson's King Kong remake will nenver rival the hilarity of K-Y's remake. Funniest late night ad, ever.

  • Contact? (Score:4, Funny)

    by jridley ( 9305 ) on Tuesday October 11, 2005 @12:58PM (#13766082)
    Hmm, how to get them on my SETI team....

  • Nonsense Statement (Score:3, Insightful)

    by deadline ( 14171 ) on Tuesday October 11, 2005 @01:00PM (#13766097) Homepage
    boosting its power by 50 per cent to create a supercomputer with the equivalent power of nearly 15,000 PCs

    Such statements are utter nonsense. First, 15,000 PC's - what kind of PC? (dual core AMD, I think not). Second, how do you measure power ? (is this for their applications, or some other metric) If they ran the numbers they would find the cluster rather typical - unless there is more to the story.

    Yes they have a a lot of processors, however, lots-o-processors != supercomputer

    • It's that dumbed-down unit of measurement.

      ...to create a supercomputer with the equivalent power of nearly 15,000 PCs.

      Or 7 Volkswagens and 3.4 Libraries of Congress...

    • Yes they have a a lot of processors, however, lots-o-processors != supercomputer

      Out of curiousity, what does make a supercomputer then? That statement, combined with your URL, intrigued me.

      • Re:Nonsense Statement (Score:3, Interesting)

        by deadline ( 14171 )
        A good question. I believe at one point Gorden Bell said "anything with 6 zeros in the price".

        The lines have blurred due to clusters. My definition is "a collection of hardware that provides a non-trivial level of performance on a single problem" Of course, "non trivial" has various interpretations. And, working toward solving a single problem is important. Rendering is a trivial parallel application as it is really a bunch of small independent problems. Most supercomputer applications would probably run

        • My definition is "a collection of hardware that provides a non-trivial level of performance on a single problem"

          OK, so a "lot of processors" could be a supercomputer, depending on the software running. And, apparently, the presense of InfiniBand. :)

          I'll agree that rendering is embarrassingly parallel.

          (I've been poking around ClusterMonkey this afternoon. I work at the Ohio Supercomputer Center @ Ohio State, and am currently implementing a SVM classifier in MATLAB to run on a Linux cluster.)

  • I guess they'll live up to their promise of amazing gorilla hair... one blade server for each hair rendering...
  • In other news, New Zealand is accused by the ONU for not respecting the Kyoto treaty.
    It appears that New Zealand is now the World n1 heat producer, the origin of that heat is currently unknown.

    Well, joke aside, I hope for them than the clim won't break...

  • The massive power of creating digital realism (Score:5, Informative)

    by digitaldc ( 879047 ) on Tuesday October 11, 2005 @01:13PM (#13766208)
    The amount of power that is needed to create a realistic outdoor scene with multiple actors is simply astounding. King Kong will most likely be candy for the eyes when it is done. Halo, the next Peter Jackson movie, will probably just as amazing.

    An interesting article on building a digital animation studio (IBM) is here:
    http://www-128.ibm.com/developerworks/library/wa-a nimstudio1/ [ibm.com]

  • Given the high degree of parallelism and the social aspects, you'd think that distributed computing would be ideal for hollywood rendering, given that you could implement sufficient security restrictions. (Security restrictions which should be perfectly managable.) How many people out there do you think would like to be able to say "I rendered part of this movie!"

    There are some issues, of course, but it strikes me as worth exploring.

    • Not many. The MPAA is already charging $7-12 for me to watch the movie plus another $10-30 if I want to buy it. The last thing they're getting from me is a single cycle of my CPU. If they send me a movie ticket they can use my network, otherwise they can put the billions of dollars they're making to good use.
    • They should have done that with Serenity. The next movie that fans demand be made, offer to finance, after Hollywood cancels the project, that gets partially produced (eg. SFX/CG) produced by those global fans, will turn the tables on Hollywood. Once the distributed consumer community figures out how to replace the "distribution deals" by which Hollywood controls global entertainment, we'll finally be into the new century.

  • Sweet! (Score:3, Funny)

    by Black Francis ( 694247 ) on Tuesday October 11, 2005 @01:33PM (#13766382)
    The IBM Xeon blade servers, each with two 3.4 gigahertz processors and 8 gigabytes of memory...
    ... now they might even be able to run Vista!
  • The IBM Xeon blade servers, each with two 3.4 gigahertz processors and 8 gigabytes of memory

    *drool*

  • The movie will still be a train wreck (Score:3, Funny)

    by kindbud ( 90044 ) on Tuesday October 11, 2005 @02:07PM (#13766638) Homepage
    I mean, come on. Why King Kong? Was the world really lacking yet another King Kong adaptation? This movie will make Gozilla look like Independence Day.
  • Too bad no-one's ever thought of using the fastest CPUs in these clusters and really doing something useful instead of using a huge number of weak CPUs to gain publicity.

  • xeons? (Score:4, Informative)

    by ruiner5000 ( 241452 ) on Tuesday October 11, 2005 @02:23PM (#13766774) Homepage
    Did Intel do a massive discount? The standard is Opteron for render farms and for good reason. Much faster, much less power, and a superior upgrade path. 150 watts for the fake dual core Xeon, or 89 for a dual core Opteron. Hmm, tough choice there. Particularly when you look at the new Spec numbers for the dual core Xeon it is a massacre with Opteron well ahead still.

    • Re:xeons? (Score:3, Interesting)

      by rodgerd ( 402 )
      Meaningless comment unless you know how they've optimised their code and practises. It may be they have a bunch of optimised render code that works well with the Xeon and would need to be re-written for the Opteron.
  • Weta Digital Grows Cluster

    They can just snip those off in an outpatient procedure now.

  • Available for hire eh? Uh yeah...lemme just pull out my checkbook ;-).
  • Given that there are about 35 people in the world who haven't seen one version or another yet, how worthwhile is any King Kong remake? I can't wait for Jackson to move onto another project like The Mote in God's Eye or the Foundation trilogy.
    • Given that there are about 35 people in the world who haven't seen one version or another yet, how worthwhile is any King Kong remake? I can't wait for Jackson to move onto another project like The Mote in God's Eye or the Foundation trilogy.

      It's not as if it's some desperate idea from the studio. Jackson has wanted to remake King Kong since he was a kid. He tried to make it before LOTR - but Miramax yanked it because there were too many monster movies that season. Now he gets to make it with a much big

  • Time to burn a little karma :P

    Korgan better watch out for RMS and GNU/flunkies. He called it: "Red Hat version of the open-source Linux operating system". Not: Red Hat version of the Free as in speach GNU/Linux operating system.

Slashdot Top Deals

I tell them to turn to the study of mathematics, for it is only there that they might escape the lusts of the flesh. -- Thomas Mann, "The Magic Mountain"

Close