Forgot your password?
typodupeerror
Microsoft IT

Debugging Microsoft.com 511

Posted by ScuttleMonkey
from the belly-of-the-beast dept.
teslatug writes "Channel 9 has an interesting video interview with Chris St.Amand and Jeff Stucky who test and debug Microsoft.com. They reveal some of the big problems they used to face such as recycling processes every 5 minutes due to memory leaks and 32 bit limitations, and being unable to push more than 10 Mbits of data to their datacenters due to Windows' networking stack limitations."
This discussion has been archived. No new comments can be posted.

Debugging Microsoft.com

Comments Filter:
  • What the... (Score:5, Funny)

    by BrainInAJar (584756) on Monday December 05, 2005 @08:28PM (#14189917)
    WMV? You serious?

    How the hell am I supposed to watch that?
    • by Trigun (685027)
      Give us a little hint. What are you using to browse slashdot right now?

      Maybe if you gave us some particuars, we could help.
      • by xQx (5744) on Tuesday December 06, 2005 @01:47AM (#14191407)
        Yeah, seriously... No, Just let me get this straight, because I'm slow...

        You are complaining that a Video about Microsoft.com, featuring Microsoft employees, created by Microsoft is released in Microsoft's favorite media format which plays natively in Microsoft Windows Media Player.

        Umm yeah, okay, it's really a secret ploy to give you linux nuts another reason to re-compile your kernel with evil capitalist codecs in it (or some other bullshit rant).
    • Easy. (Score:4, Informative)

      by wilymage (934907) <wily@@@bur...st> on Monday December 05, 2005 @08:31PM (#14189953) Homepage Journal
      1. mplayer [mplayerhq.hu]

      2. xine [xinehq.de]

      Not that tough, really, now is it?
      • Re:Easy. (Score:3, Insightful)

        by GNUALMAFUERTE (697061)
        Those are players, the codecs are still proprietary and binary only.
      • Re:Easy. (Score:5, Insightful)

        by AstroDrabb (534369) on Monday December 05, 2005 @09:24PM (#14190275)
        You listed *players* not codecs. I will assume you meant one of the players you listed with the binary win32 codecs that usually get installed under /usr/lib/win32. They do work well and I use them. However, they are basically 32-bit x86 only, so if you are not running 32-bit x86, you are SOL. Maybe the GP is running PPC Linux or a 64-bit Linux? Or maybe the GP doesn't want to run binary only win32 dll files on his computer?
        • Re:Easy. (Score:4, Insightful)

          by Lisandro (799651) on Monday December 05, 2005 @10:11PM (#14190512)
          And even then, both xine and mplayer fail to play some newer formats, like the latest version of WMV. Which is a shitty format to begin with, but it's all arround the internet it seems...
        • Re:Easy. (Score:3, Informative)

          by evilviper (135110)
          However, they are basically 32-bit x86 only, so if you are not running 32-bit x86, you are SOL. Maybe the GP is running PPC Linux or a 64-bit Linux?

          32-bit DLLs work fine on 64-bit x86 Linux. You have to compile MPlayer as a 32-bit program, of course, but you're still running it on a 64-bit processor, and a 64-bit Linux OS.
    • by Tackhead (54550) on Monday December 05, 2005 @08:41PM (#14190027)
      > WMV? You serious?
      > How the hell am I supposed to watch that?

      Well, if you're not running Windows, how the hell else are you supposed to get memory leaks? They don't just grow on B-Trees, y'know!

    • Same way as I did, with mplayer
    • by alfrin (858861)
      Well do you think want to give us Linux users the satisfaction of seeing Microsoft employees admitting faults in their software?
  • Missing info... (Score:5, Informative)

    by DaHat (247651) on Monday December 05, 2005 @08:28PM (#14189921) Homepage
    The summary is missing the fact that many of their problems went away after upgrading to an early 64 bit version of Vista with its improved networking stack.
  • by Anonymous Coward
    The next post will be about debugging slashdot.
  • by Anonymous Coward on Monday December 05, 2005 @08:31PM (#14189944)
    Why don't they just migrate to Apache on OpenBSD? :)

    Oh, right...
  • Hmm.... (Score:5, Funny)

    by lord_sarpedon (917201) on Monday December 05, 2005 @08:34PM (#14189976)
    I suppose that, transitively, it is due to a limitation in an archaic version of the BSD stack.
  • Suprised? (Score:3, Funny)

    by MasterPi (896501) on Monday December 05, 2005 @08:34PM (#14189983) Homepage
    Is anybody really suprised here? What they didn't tell us is that there's a top-secret Debian redundancy server running behind it just in case all hell breaks loose. Nothing to see here, move along.
  • The Desk (Score:2, Funny)

    by techsoldaten (309296) *
    Did anyone notice the desk? What kind of soda cans are those on his desk? I thought they were Hansens, but it does not look like any I have ever seen.

    M
    • The Video (Score:3, Insightful)

      by TubeSteak (669689)
      The video is 38 minutes long
      http://wm.microsoft.com/ms/msnse/0511/25766/micros oft_dot_com_debug_team_2005_MBR.wmv [microsoft.com]

      While I usually RTFA (unlike most slashbots) I think we can all agree that at 40 minutes maybe 1/2 a percent of /.ers will actually watch this.

      /me waits for the transcript

      And yea, I saw the cans, but the bit-rate of that video is so low, I have no clue what they were. Maybe that red one on the left is a coke or dr. pepper?

    • Re:The Desk (Score:5, Informative)

      by Procyon101 (61366) on Monday December 05, 2005 @10:41PM (#14190668) Journal
      The one on the left is Coke, the other 3 are Red Talking Rain. Personally, I'm a Green Talking Rain programmer, but I can respect teh other side :) Talking rain (particularly green) is the nectar of the programmers here in Seattle.

      You see, Microsoft started the great thing a few years back where every floor was stocked with 2 giant refrigerators of free soda. The rest of the local software companies quickly moved to copy this ingenious move, so you can't program and not be in contact will all the free soda you can drink. This sounds pretty cool until you've done it for about 2 years. At that time, assuming you are not a natural soda addict, the last thing on earth you want to drink is any kind of beverage with sugar in it, because you are so unbelievably sugared out. In come Talking Rain. Talking Rain is a simple carbonated spring water, with just a hint of fruit oil added, and no sugar. Green Talking Rain adds lime oil, and Red Talking Rain adds Rasberry, I think, although being a Greener myself, I never really paid attention. The fact that only senior programmers have completed this Talking Rain pupation, allows you to easily glance at someone's trash can in their office and peg them for a Senior or Junior level developer. You will almost never see a Junior level developer drinking Talking Rain, and almost never see a Senior level NOT drink it. Kind of a free soda pecking order.

      Of course I may be reading to much into this, but my Greener roots run deep :)
  • Ironic? (Score:5, Funny)

    by heistgonewrong (808413) * on Monday December 05, 2005 @08:38PM (#14190005) Homepage Journal
    Is that not one of the most ironic things you've ever heard? The limitations of the operating system made by the same company holding back another division? Shock and awe.
  • by ThinkFr33ly (902481) on Monday December 05, 2005 @08:42PM (#14190037)
    The limitations discussed in the video of the Windows TCP stack are not limited to Windows. These are limitations imposed by a to-the-spec implementation of TCP. TCP is 30+ years old, and it wasn't designed for the kinds of networks it runs on today.

    The new TCP stack in Vista effectively implements TCP is such a way that it removes these limitations while preserving compatibility with old stack implementations.
    • I'm really confused. I was under the impression that any old implementation of RFC 793 [isi.edu] qualifies as "TCP".

      In other words, TCP is a protocol, not an algorithm.

      So ... if Vista has some fabulous new algorithms for implementing TCP, then why can't other OSes be patched to benefit from those algorithms also? OR, if Vista is implementing something other than TCP, then how can it be (fully) backwards compatible?

      Seems like the word "compatibility" might need to be scrutinized here.

    • What "modern" OS still runs a TCP stack as it was created many, many moons ago?

      TCP has evolved quite a bit over the last 30 years, and new RFCs and other standards are constantly enhancing and obsoleting older versions of the standard.

      You seem to imply that an implementation built today "to-the-spec" would be built against on some 30-year-old draft and design. Today's TCP standards (which include a number of "experimental", "optional", "designed-for-high-latency" etc extensions), however, are quite capable
      • Read this [microsoft.com] to see what they're doing.

        I agree that no modern OS runs a 30 year old stack... but most modern OS's today still have major issues with high latency connections even when those pipes have plenty of bandwidth. There is nothing we can do about a 100ms latency when the connection is 5000 miles long, but there is a lot we can do to improve the TCP protocol to optimize for those long distance/high bandwidth connections that are becoming more and more common.
  • by iamwoodyjones (562550) on Monday December 05, 2005 @08:49PM (#14190082) Journal
    Interviewer: "Hey dude."

    Chris St.Amand "What up bro"

    Interviewer: "So like what happened when you worked on microsoft.com? Oh but first...Did you get all the chicks at the bars when you mentioned your job or what?"

    Chris St.Amand "Oh totally. I'd just say, 'what up babe. I work on the microsoft.com web portal' and she'd degfrag my harddrive all night."

    Interviewer: "Sweet. So what was your biggest hurrdle writing all that HTML? After all that's a complicated langaguage to master."

    Chris St.Amand "It'd definelty have to be that F'ing page not found shit. You don't know how many times I'd go to microsoft.com after doing a big update and it'd just say four-oh something and the page just wouldn't show up. You know we tried to put up a 420 page not found but got in trouble with our boss."

    Interviewer: "Yea totally! That would have been cool. Oh ummm let's see here. So what other problems did you have?"

    Chris St.Amand: "Not being able to use FreeBSD to serve that shit. When I first heard I actually had to use Microsoft I was completely like, 'Not cool Bill. Not F'ing cool, Bill.'

    Interviewer: "Any thing else? Like was it hard to get up every day in the morning knowing that your existence was updating microsoft.com HTML?"

    Chris St.Amand: "Yea I tried sucicide a number of times. But then I discovered that I could just completely make up new HTML tags and that was a lot of fun."

    Interviewer: "Make up HTML?"

    Chris St.Amand: "Oh yea, we're microsoft. When I first started they told me that no other browsers exist other then that big blue F'ing E and that no other operating systems exist. And that I could do whatever I wanted to do. So I just started making up *ALL KINDS* of crazy ass HTML.

    Interviewer: "Cool dude. You rock. Anything else you want to mention?"

    Chris St.Amand: "Yea you know all that crazy F'ed up HTML that all of our products output? You know without indention and messed up question marks everywhere? That was me. I was all hung over the day I added that. And that's about it."

    Interviewer: Thanks Chris, I'm sure you'll go down in infamancy for such a piece of F'ing shit web page and end up in some lame ass 'Don't write web pages like this' hall of fame.

    Chris St.Amand: "Peace out and remeber to eat your greens not smoke 'em!"
  • by GNUALMAFUERTE (697061) <.almafuerte. .at. .gmail.com.> on Monday December 05, 2005 @09:00PM (#14190147)
    They should be redesigned.

    That's a big problem of software made by companys:

    1 - The company's cashflow is based arround selling new versions of the software
    2 - They can't sell to it's customers improvements that they customers can't see
    3 - There is a fixed time that can go by beetween one release and the next one
    4 - Resources are limited

    Because of this, a major redesign is something that won't be profitable, because only the advanced users will note the changes, but 99% of their customers won't, so the software won't sell well. Bug fixes also won't sell, because they are also unvisible to the naked eye of the majority of the userbase, and also customers expect those changes to be free.
    So, some companys only can expect revenue from a given software once a year, and they have to invest into that software, a given set of limited resources over, say, 6 months, when they have to freeze the featureset so they can start debugging. Seeing which things sell, they will obviously focus their atention on: New Features, and a nicer GUI.
    OTH, a project that doesn't have a company running it, can just get out lots of upgrades, when needed, and focus their time on making the software better, even if some of the changes made to the software won't be seen by most of it's users.

    With software prices dropping, and Free Software proving to be a better option, the budget of software companys will be even more limited, and we won't see this situation changing anytime soon.
  • Compound TCP (Score:3, Informative)

    by kyoko21 (198413) on Monday December 05, 2005 @09:07PM (#14190192)
    Slightly off topic, but the new Windows TCP stack will be implementing their new Compound TCP stack, aka, CTCP. More information can be read here:

    http://research.microsoft.com/research/pubs/view.a spx?type=Technical%20Report&id=940 [microsoft.com]
  • Err ... (Score:5, Funny)

    by ggvaidya (747058) on Monday December 05, 2005 @09:12PM (#14190212) Homepage Journal
    Am I the only one who looked at the title and thought: "debug microsoft.com? Who still uses .com files any more?"

    Yup, thought so. I suck.
  • by Anonymous Coward on Monday December 05, 2005 @09:13PM (#14190217)

    They reveal some of the big problems they used to face such as recycling processes every 5 minutes due to memory leaks and 32 bit limitations, and being unable to push more than 10 Mbits of data to their datacenters due to Windows' networking stack limitations."

    Micro$oft needs 64 bit so it can leak more memory faster and stay running. Or at least this is how I read this.

    As for 10mbs, maybe they should put a Linux/BSD/UNIX cache in front of those servers like MSNBC did to get through the last olympics.

  • by E-Lad (1262) on Monday December 05, 2005 @09:41PM (#14190368) Homepage
    Wow. Just wow.

    I look at Solaris (err, OpenSolaris) and how it can now push a 10Gb/s interface at line speed [sun.com] (or close to it) and MS has struggled up until recently to get satisfactory speeds above 10Mbit/s ?

    Yet another "how do users/admins accept this as OK" thought going through my head re: Windows internals.
    • by ThinkFr33ly (902481) on Monday December 05, 2005 @10:05PM (#14190491)
      Watch the video. That wasn't the problem.

      The problem was connecting two datacenters that were physically seperated by a long distance but connected with a high bandwidth pipe... the TCP protocol has problems with this because of latency issues.

      Read this [microsoft.com] to see how they solved it.
      • #1: tcp window sizes are neat. By changing a couple proc variables, I can push 350mbps with a single tcp flow over a busy I2 link with a single stock fedora core 3 kernel, on a 1ghz P3.

        #2: -10 points for using "synergy" in technical info (linked ms.com article)
    • What does a TCP/IP improvement to help high-connection-count low-latency scenario have to do with MS trying to solve the problem of a VERY high latency, single connection scenario?

      Or are you just replying to the 3 sentance summary without any information or knowing what you're talking about?

      Microsoft's problems were with the TCP specifications, which they adhered to TOO closely. From the paper, NOT specific to windows (specific to any fully compliant TCP implementation), "under a 10GBPS link with 100ms del
  • Remember Hotmail? (Score:5, Interesting)

    by Anti-Trend (857000) on Monday December 05, 2005 @09:47PM (#14190400) Homepage Journal
    That wasn't the only time that MS has had to eat humble pie about their server capabilities. Remember the whole Hotmail fiasco in late '97 when Microsoft acquired it? The whole thing was running on UNIX and ran just fine. They tried to replace it with NT servers, and it just couldn't stand up under the weight no matter how much hardware they threw at it. As a result, they had to stick with UNIX for quite a while until they could get Windows to the point where they could even pretend to make any real use of it.

    The following is just hearsay, as I've never actually worked for MS. But a couple of engineer buddies I used to work with did some subcontracting for MS, and they said they deployed a whole lot of internal-facing *nix servers during that period. I tend to believe it, because the MS security guys who taught some seminars I attended wouldn't confirm or deny that they used any Linux internally. If they could have denied it in clean conscience, wouldn't they have done so emphatically?

    • Re:Remember Hotmail? (Score:3, Informative)

      by robogun (466062)
      Remember the whole Hotmail fiasco in late '97 when Microsoft acquired it? The whole thing was running on UNIX and ran just fine. They tried to replace it with NT servers, and it just couldn't stand up under the weight no matter how much hardware they threw at it.

      It still can't stand up to the weight. Have you tried using Hotmail in the middle of the day and get those SERVER TOO BUSY errors? If it even responds!
  • by Jonboy X (319895) <jonathan@oexner.alum@wpi@edu> on Monday December 05, 2005 @09:52PM (#14190427) Journal
    Hmm, nearly-direct link to a 145-megabyte video file on the /. front page, posted right as the geeks of the world are getting home from work. What are you, crazy? Are you trying to Slashdot Microsoft?

    Don't answer that.
  • by erroneus (253617) on Monday December 05, 2005 @10:26PM (#14190595) Homepage
    You could scan through all of my old posts for background if you like, but back when NT 4.0 was brand new, I helped to save a failing ISP (for at least the next 6 months or so) by setting up a new mail server to replace the one that was failing ever 2 to 10 minutes. I used a machine with less than half the power and resources of the machine already running... and loaded slackware. I think the kernel was jsut over 1.00 at the time.

    Yeah, "old technology" couldn't do anything better than new stuff like NT right? Come to think of it, there's not a LOT of difference between XP's kernel and NT's from what I understand... a few bug fixes here and there... but basically, it uses the same vulnerable messaging scheme and drivers running at ring-0 and all that. ...I guess I've repeated enough digs on microsoft for one posting...
    • Yeah, "old technology" couldn't do anything better than new stuff like NT right? Come to think of it, there's not a LOT of difference between XP's kernel and NT's from what I understand... a few bug fixes here and there... but basically, it uses the same vulnerable messaging scheme and drivers running at ring-0 and all that. ...I guess I've repeated enough digs on microsoft for one posting...

      Drivers generally run in kernel mode in Linux, and most other operating systems for that matter. One of the few that
  • by sheldon (2322) on Monday December 05, 2005 @11:29PM (#14190857)
    Slashdot has turned from "Microsoft sucks" to waxing poetically about how Microsoft used to suck.

    How times change...
  • by adventuregeek (128208) on Monday December 05, 2005 @11:48PM (#14190945)
    It was one of there secondary sites, something like blah.microsoft.com. The ISP was supposed to be hosting it on a colo NT box as part of an outsourced hosting contract. Well the site crashed constantly and the support team got sick of the late night pager calls and moved it over to a BSDI box with Apache and spoofed the server headers to read IIS, never told the M$ guys.
    • Pardon me if I think you're lying through your teeth. How could they not notice that they're no longer connecting to a Windows server? They would still have to connect via FTP or something other protocol, did you spoof those too? Not just that, how did you manage to fake the whole directory tree? If they connect to upload files, they'd notice it was a unix system by the file hierarchy and the fact that ASP DIDN'T WORK ANYMORE. Yes, there are some *nix ASP products, but they don't work that well. They'
  • by dilvish_the_damned (167205) on Tuesday December 06, 2005 @12:51AM (#14191188) Journal
    It gracefully "cycles" your process so you have your memory leaks. If only other apps were coded for memory leaks.
  • Kudos to microsoft (Score:5, Insightful)

    by urlgrey (798089) * on Tuesday December 06, 2005 @01:10AM (#14191271) Homepage
    Microsoft--and the two staffers shown in this video--deserve strong praise for the *unedited* candor, the self-depricating humor, and the absense of spin on this video.

    Maybe I've missed the comments, but what no one seems to mention here is that these guys--clearly both geeks at heart (in a good way)--really are peeling back a lot of the layers of MS's site. The candor about their security problems, the 2gb memory issues, and a variety of other things was refreshing.

    Heck, they even mention firefox. :-)

    Good work all. Good work.

  • by karups2 (611529) on Tuesday December 06, 2005 @02:05AM (#14191466)
    At around 10:25 in the video Chris St. Amand, who runs Microsoft's website and data center, types in his password, which the camera recorded. And the video is hosted off of Microsoft's website...although I don't know how long that'll still be operational.

Uncompensated overtime? Just Say No.

Working...