The Setup Behind Microsoft.com

Toreo asesino writes "Jeff Alexander gives an insight into how Microsoft runs its main sites. Interesting details include having no firewall, having to manage 650 GB of IIS logs every day, and the use of their yet unreleased Windows Server 2008 in a production environment.

Mostly how they run it

is have some crazy sys admins throw chairs around.

Re:Mostly how they run it

With Microsoft Windows Server 2008, chairs practically throw themselves!

Beta in production environment.

"Windows Server 2008 in a production environment."

So even MS has given up on Vista.

Re:Beta in production environment.

Vista was never meant as a server. Same as XP isn't used as a server, it's Server 2003.

Re:Beta in production environment.

Funny, but you're wrong. Pro is for networking enviorments where you need RDP, policies, ability to join a domain, file encryption, etc. Home lacks these.

Re:Beta in production environment.

Tis a sad day when the fanbois can't even get their insults right. shameful.

Re:Beta in production environment.

And of course it's already been modded up (at least only as funny). To clarify why the GP is wrong, from the wikipedia entry [wikipedia.org] on Windows server 2008:

Windows Server 2008 introduces most of the new features from Windows Vista to Windows Server. This is a similar relationship to that between Windows Server 2003 and Windows XP.

Gotta give credit to MS for eating their own dog food...

Allow incoming connection on port 80? Confirm/deny

Re:Beta in production environment.

Program WinYes! is trying to perform an action on a dialog box. Allow/Deny?

Re:Beta in production environment.

Which we do on a regular basis. Every few weeks I see emails going around from higher-ups asking us to test their team's RC or beta stuff at home for them, and the project I'm working on has been dependent on VS2008 since beta2. Everyone here has their favourite project they like to keep tabs on. I've got longhorn server 2008 running on one of my machines here.

That said, the choice to use longhorn server in production isn't actually a bad one. It's really, REALLY stable. I keep hearing (from people both inside and outside the company) that it's more stable than 2003 is (and 2003 has the benefits of multiple service packs). It's also a lot more configurable about what it runs, and how much of it it enables when it's installed. I wouldn't bet the entire stable on it, but I'd be willing to put money on it getting a place.

All in all, it's pretty sweet, if you look at it from the sysadmin perspective. Also, the stuff you can setup when you couple it with vista is really nice (from a security standpoint, particularly). That said, some of that functionality is being backported to XP with SP3 or whatever.

Re:Beta in production environment.

Ok, but is the OS *still* organized like crap? I mean, is C:\Windows still a dumping ground for a bunch of arbitrarily named data files, log files, drivers, and libraries using, for the most part, the old 8.3 naming convention?

Dude, if you can't hack that right now, how are you dealing with unix instead?

If any platform's based on a standard of bizarre naming due to space saving stupidity, that's it. Far more so than windows. Infact, name any mature platform that's based on reasonable standards for it's underlying API's and structure.

Didn't think you could. While it's true that things like the FHS are helping on the unix side, try telling an oldschool developer like oracle that they need to follow it. They'll laugh. and laugh.

and laugh.

Windows is in much the same position. At least .NET has made this significantly less painful, because it was considered ahead of time (it's not much easier to actually manage, but that's the tools more than anything, and just takes a bit of experience.... which unsurprisingly, is what dealing with the idiosyncracies of the old systems take anyway!)

ash

Re:Beta in production environment.

Dude, if you can't hack that right now, how are you dealing with unix instead?

Because at least Unix has conventions.

If any platform's based on a standard of bizarre naming due to space saving stupidity, that's it.

Really? Ok, lets open up C:\Windows on one of our Windows servers. Hmmm a folder named "$hf_mig$". I suppose you know what that means or what convention that follows? Or C:\Windows\adam. Kinda looks like it might be some directory tools. Maybe ADAM = Active Directory AdMinistration? What's that doing there anyway? I could keep going down the list. I suppose there is a very good reason why there are .BMP files in C:\Windows? Desktop wallpapers? Come on. I wonder if they're related the other brilliantly named files such as SET2.tmp and SET3.tmp in that same directory. And don't get me started on the insanity that is C:\Windows\System32. Hardly a single file/folder that doesn't use 8.3 naming. I haven't clue what have that stuff is doing there.

Infact, name any mature platform that's based on reasonable standards for it's underlying API's and structure.

First of all, I was only talking about superficial organization. And if you want to see something nice, have a look at OS X some time. Not only is the System (/System) well organized, but most applications are neatly self contained in /Applications/Some.app. They usually don't spew files all over the place when installed. You know where the term DLL Hell comes from, don't you?

Didn't think you could. While it's true that things like the FHS are helping on the unix side, try telling an oldschool developer like oracle that they need to follow it. They'll laugh. and laugh.

I could give fuck-all what Oracle thinks. My Debian systems are very well organized, thank you very much. I don't find desktop wallpapers in /usr/lib. I don't find temporary files for applications in /usr/bin. FreeBSD is even cleaner. The system files never change unless I explicitly do an upgrade. All supplementary software (ports, mostly) goes in /usr/local. With Windows, on the other hand, who knows what strange and wonderful new files I might find dumped in C:\Windows tomorrow. Maybe $hf_mig2$. WHich would be version 2.0 of whtever that is, i guess.

-matthew

Re:Beta in production environment.

Because at least Unix has conventions.

Conventions are a nice way of saying "that's the way it's always been, so that's the way it stays." Windows has similar problems left over from legacy, going all the way back to CP/M. Yes, this sucks, but so does some conventions in unixland. Just ask a Solaris 10 admin how much it sucks when your upstream vendor breaks decades-long convention.

Really? Ok, lets open up C:\Windows on one of our Windows servers. Hmmm a folder named "$hf_mig$". I suppose you know what that means or what convention that follows? Or C:\Windows\adam. Kinda looks like it might be some directory tools. Maybe ADAM = Active Directory AdMinistration? What's that doing there anyway? I could keep going down the list. I suppose there is a very good reason why there are .BMP files in C:\Windows? Desktop wallpapers? Come on. I wonder if they're related the other brilliantly named files such as SET2.tmp and SET3.tmp in that same directory. And don't get me started on the insanity that is C:\Windows\System32. Hardly a single file/folder that doesn't use 8.3 naming. I haven't clue what have that stuff is doing there.

You're not looking in the right place. Microsoft, love it or hate it, worked out a long time ago that 'filename' and 'metadata' aren't necessarily the same thing. The filename and path are just handy locational indexes, and don't necessarily need to mean *anything*. Sure, a DLL can, and often, for newer stuff, IS far longer than 8.3, but it wasn't until later versions of NT (3.5/4.0, I don't remember my history too well) that support for it kicked in well enough, and there's some legacy stuff around. You don't break legacy just because it's fun. Microsoft gets this right, even if they had to tread over it a fair bit in vista, and add some nasty hacks to deal with most of the fallout.

Anyway, as I was saying, you're not looking in the right place. Case study: C:\windows\system32\apss.dll: Microsoft(r) InfoTech Storage System Library.
Problem solved. (it's not at all difficult to use something like powershell (or possibly other tools) to just print this out in a souped up version of ls with a little scripting, I might add, just like I can do a few similar scripting tricks on my debian system to tell you who owns the copyright to 90% of .so's in /usr/lib.)

Want another one?

c:\windows\System32\bitsigd.dll: Background Intelligent Transfer Service IGD Support

Oh look, another one, fully named.

Of course, this starts to fall down when the file doesn't contain metadata, but that's a problem for, say, XML schema files in /usr/share/ on linux too. The organisation might be a bit better, but not by much. The saving grace there is that I have dpkg to work shit out for me. .NET goes even further. You can register as many different versions of a namespace as you like, and .NET will do the mapping for you if you request a specific version.

First of all, I was only talking about superficial organization. And if you want to see something nice, have a look at OS X some time. Not only is the System (/System) well organized, but most applications are neatly self contained in /Applications/Some.app. They usually don't spew files all over the place when installed. You know where the term DLL Hell comes from, don't you?

Yes. I do. .NET does a good job of solving this quite nicely. Adds public/private keys into the mix too, plus a bunch of other mechanisms. .NET isn't just for C# either. It deals with VB, C++, and (ahahahha) J# too.
I will admit that the mac platform is neatly arranged, but their QA seems to have gone to the toilet right now. A place that windows' QA has emerged from rather nicely, I should mention.

As for random stuff appearing in random places, try dealing with commercial software. Even on linux, the developers will put shit in strange places. Open source software is a different matter, you've got enough control that you, or the maintainers, can apply the shoe-horn. Windows doesn't have this problem either. Windows software goes in where it should, and, i should mention, is *legally* obliged to go away completely cleanly when the user requests. I'm not kidding about this. We do a lot of QA just making sure that 'uninstall' for our newer shit works.

We can't be responsible for what third parties do, however. Neither can apple (I just *love* dealing with adobe's software on apples, btw. Or Zend Developer Framework. mmmhm. ) Nor you. Install maya on linux sometime. Or matlab, or something else that you can't fuck with the organisational structure of, because the licensing server would crack the shits.

ash

Re:Beta in production environment.

Windows Server 2008 is (or rather, will be) effectively "Windows Vista Server Edition", just as Windows Server 2003 is effectively "Windows XP Server Edition".

Firewall Schmirewall

No firewall? Of course not!

Microsoft servers are notorious for their invulnerability.

Re:Firewall Schmirewall

Of course they have a firewall, just watch the difference between a tcptraceroute to a public port (like 80) and tcptraceroute to the same ip but some other port (like 110 pop3 for example). You'll see that packets get dropped at some point indicating a firewall. It's not a RST (port closed) it's just dropping packets for nonpublic services. That is a packet filtering firewall.

But generally..

Router ACLs are in place to block unnecessary ports
Cisco Guards for DoS detection and automated response

In other words, they don't use firewalling where you have administrator defined rules to control traffic flow, they use networking equipment that accept administrator defined rules to control traffic flow .... totally different..

What in the world do *you* perceive the difference being between a 'firewall' and a router blocking ports based on source and destination being compared with a set of rules (aka ACLs)? Generally, firewall rules *can* get more complex than that, but mere port blocking by an intermediate router has been considered a firewall, even if it doesn't log violating or accepted packets, even if it doesn't have complex rules about connection state. Even if it doesn't have the word 'firewall' emblazened on the chassis somewhere.

Re:But generally..

The distinction between port filtering + ACLs and today's notion of "firewall" that's actually useful is of a stateful firewall, doing stateful packet inspection, with policies based on not just the packet you're picking a TCP header out of. If you tried to sell a stateless filter as a "firewall" today, you'd be laughed out of the market.

And no, I don't see any need to firewall a web farm either.

Re:But generally..

No, because you'd have to go to considerable effort to configure it in such a way that what you say would actually happen. Hell, even my Windows Server 2003 machine is still running stable and virus/spyware free after about five years (or so).

Re:Firewall Schmirewall

Actually you're wrong. They're blocking ports. Port blocking != firewall.

Ah, the little children. Do you know what the first firewalls were? Routers with access lists. Anything that blocks anything from going to one place from another is a firewall. Port blocking is a firewall, and there exists no firewall I know of that can't be configured to do nothing other than port blocking. You don't have to inspect packets, track flows, or any of those other things to be a firewall, all you have to do is offer some means of restricting traffic. And blocking ports does that.

Re:Firewall Schmirewall

from the article:
"...At this point we still don't use firewalls for MS.COM..."

and then

"Router ACLs are in place to block unnecessary ports"

blocking unnecessary ports is a firewall feature (IMHO ?)

Anyway it looks quite impressive. I still don't understand how to handle 650 GB of logs :-).

Re:Firewall Schmirewall

Well, remember the story a while back about MS using Linux for some things? I think we just found where they use it. Storing their logs in /dev/nul is the most likely way they deal with 650 GB of logs.

Re:Firewall Schmirewall

Storing their logs in /dev/nul is the most likely way they deal with 650 GB of logs.

Well geez.. in that case I sure hope they do regular backups of /dev/null! ;-)

Re:Firewall Schmirewall

$ cat /dev/null | gzip - > devnull.gz
$

Works fine for me. Are you sure you're not confusing /dev/null with /dev/zero? The latter's a real bitch, it's always too large for my destination drive! Gzip helps though; you can get compression ratios of approximately 2000:1.

Re:Firewall Schmirewall

Large scale log processing isn't hard if you have the right tools [apache.org]. :)

Re:Firewall Schmirewall

Inventing some binary format is pointless

I'm guessing you have no prior experience with Microsoft Office then..

Re:Firewall Schmirewall

My question is why are the logs in ASCII text format? When all you want is say the IP [4 bytes], time of day [4 bytes], URI, referrer and return code [do you really care about their browser strings? You are MS after all, just assume it's IE]. Storing an IP as text requires on average 15 bytes, so right there you can shave off 11 bytes with a binary IP. Time of day is worse, a date+time string is like 25 chars. Doesn't seem like much, but multiply the 32 bytes per entry you save by say 50 million hits and that's 1.5Gbyte you saved. That's not counting the white space you can remove, and a simple huffman code you could apply to the URL/referrer.

Logging in fixed format is not more efficient than variable format text files (unless we're talking about transactions but we're not). Let's assume you're logging the basics: IP address, Timestamp, Return code, URI and we'll look at logging in fixed format then variable format.

[abcd] [timestmap] [code] [URI]
4 bytes 8 bytes 1 byte 50 bytes (you actually need 2 bytes for HTTP return code, but let's ignore that)

Every record will require 63 bytes and we'll round up to 64 for proper word alignment). So, if we log 1000 messages, we will consume 64,000 bytes total.

Ok. Now for text logging with space delimiters. We have 3 options below, each requiring slightly less space than the previous. We'll run totals for each.

123.567.890.123 YYYYMMDDHHMMSS x URI...............\n
16 bytes 15 bytes 2 bytes 50 bytes 1 byte

123.567.890.123 1197572382 x URI...............\n (UNIX time)
16 bytes 11 bytes 2 bytes 50 bytes 1 byte

1235678901231197572382xURI...............\n (UNIX time)
12 bytes 10 bytes 1 bytes 50 bytes 1 byte

16 + 15 + 2 + 50 + 1 = 84 bytes * 1000 = 84,000 bytes
16 + 11 + 2 + 50 + 1 = 80 bytes * 1000 = 80,000 bytes
12 + 10 + 1 + 50 + 1 = 74 bytes * 1000 = 74,000 bytes

Wow. Fixed binary format kicks variable text format's ass. Wrong. This assumes the URI (or message) block will always occupy 50 bytes. It will not. Let's go right down the middle and assume it averages 25 bytes and we'll recalculate.

16 + 15 + 2 + 25 + 1 = 59 bytes * 1000 = 59,000 bytes
16 + 11 + 2 + 25 + 1 = 55 bytes * 1000 = 55,000 bytes
12 + 10 + 1 + 25 + 1 = 49 bytes * 1000 = 49,000 bytes

Variable text format almost always beats fixed binary format for logging. That's why Microsoft (and the rest of the world) stores log files as text. Plus, it's far easier to manage and debug when you can slice and dice the files with standard command line tools.

One more thing. I know what you might be thinking. We're logging URLS, which will probably consume the majority of the 50 byte allotment. Most developers will calculate an average width size and double it, so no matter what we'll still be filling about 50% of the message section.

Last point. If I were to use your example, the savings with text logging would even be greater. 2 URLS would be stored, both consuming about 50% of their data block. IP address, timestamp, URI, Referrer URI, Return Code. There's also a bunch of other little optimizations you can do such as storing the domain, year, month, and day in the filename rather than in the data or dropping the least significant byte in the HTTP return code.

Supporting

The highly objective and insightful article mentions, for example,

Windows and IIS...rock solid and secure!

Way to go with supporting the troops there.

Re:Supporting

How many times have you seen the microsoft.com website down / hacked?

Re:Supporting

Reliability in numbers. If you have 30 machines running your website, no one will notice if one goes down.

Re:Supporting

Whoopsie, looks like Akamai uses IIS now - I'm behind the times, I guess:

% nmap -A -T4 -F -P0 www.microsoft.com
 
Starting nmap 3.81 ( http://www.insecure.org/nmap/ ) at 2007-12-13 11:48 EST
Interesting ports on wwwbaytest2.microsoft.com (207.46.19.254):
(The 1218 ports scanned but not shown below are in state: filtered)
PORT STATE SERVICE VERSION
80/tcp open http Microsoft IIS webserver 7.0
179/tcp closed bgp
443/tcp open ssl/http Microsoft IIS webserver 7.0
 
Nmap finished: 1 IP address (1 host up) scanned in 167.891 seconds

Re:Supporting

The highly objective and insightful article mentions, for example,

"Windows and IIS...rock solid and secure!"

Talc [wikipedia.org] is technically a rock...

Microsoft brainwashing

Windows and IIS...rock solid and secure! www.microsoft.com is on Windows Server 2008/IIS7, MSDN/TechNet are migrating to Win2k8/IIS7, and update.microsoft.com is on Windows Server 2003/IIS6. We do all the normal shut-off-unused-services practices that line up with MS published security guidance and we utilize GFS images to ensure standardized builds of systems.

This guy is brainwashed. There should be no unused services turned on by default! Admins shouldn't have to shutoff unused services -- they shouldn't be enabled unless necessary. Also, rock solid and secure? Uh, didn't I read an article not too long ago about how the update.microsoft.com site was broken into?

Re:Microsoft brainwashing

You realize that Win2k3 does turn off most services by default, and Win2k8 takes this even further by not installing them at all.

Uh, didn't I read an article not too long ago about how the update.microsoft.com site was broken into?

Link, please?

Re:Microsoft brainwashing

Actually, when you first boot Windows Server it pops up with the "Configure Your Server" page, and an extra note that until you've set up roles on it, nothing will work. As in, it hasn't started IIS, it hasn't started AD, it hasn't even started Terminal Services. And until you've picked which ones you want to run, it wont even allow inbound connections whatsoever!

Re:Microsoft brainwashing

Uh, didn't I read an article not too long ago about how the update.microsoft.com site was broken into?

Link, please?

http://update.microsoft.com/ [microsoft.com]

Hi, and welcome to Bizaro World...

Interesting details include having no firewall, having to manage 650 GB of IIS logs every day, and the use of their yet unreleased Windows Server 2008 in a production environment.

Please try a complimentary goatee.

wtf!

They run AV when they can? No firewalls? It's like a 1960s flashback!

I wonder what platform they use...

...I am guessing they do not use an Apache Cluster :)

Eating dogfood is good

How can anyone complain that they're running Server 2008? My company's software quality dropped considerably when we stopped eating our own dogfood two years ago. When techs, engineers and everyone else is stuck with the same problems as the future ell-users, shit gets fixed a lot faster and a lot better.

Re:Eating dogfood is good

Not complaining in TFA, but this is /. -- I just anticipated the howls of the unwashed hordes rightfully bitching about yet another "professional" OS with a markedly unprofessional Teletubbies UI which certainly isn't ready for market yet, all while ignoring MS' internal dogfood consumption. I'll bet if enough Microsofties had eaten Office dogfood you could shut off that fucking control-click "Research" panel easily.

Nevermind that the UI for 2008 is roughly the same as 2003, only with a more extensive (yet still looking clean and fairly spartan with the eyecandy) set of configuration utilities for roles and features. Just wish I could say the same for the control panel. :)

As for the 'research' panel... okay, I work here at microsoft, and I own my own copies of office at home, and I have no idea what that is. Of course, I'm hardly an office power user.

You can bet your bottom dollar that office 2007 is all that's in use around most of the company. As is vista, although it tends to be a mixture of vista, xp and 2003/2008 in most offices, usually for a variety of legacy reasons (maintenance of older projects, testing, etc)

I've got all but XP myself, but only because I haven't needed it to do my job.

No firewalls?

If they don't have firewalls, then I have a definition of a firewall wrong.

look:

In terms of how we protect the sites, we utilize (starting at the outside edge of the network and working in):

      1.
            Cisco Guards for DoS detection and automated response
      2.
            Router ACLs are in place to block unnecessary ports ...

No a firewall, but...

FTA: "Router ACLs are in place to block unnecessary ports" While that might not provide SPI and other benefits of a true firewall, it's still a hell of a lot different than plugging a box into a wide open connection.

Priceless...

Cisco Router: ~$700
Server to run it on: ~$2000
Beta testing Microsofts new server 2008 in a production environment: Priceless

Re:Priceless...

It's called Alpha testing in this case. It's good marketing on their part to say, "We're so sure our software is good we use our pre-Beta software in a production environment." Never mind the fact that they have Server 2003 waiting ready to take over when their 2008 server horks itself.

Ever tried to bookmark something on that site?

Its like they change the URLs weekly.
I wonder if its on purpose (to avoid bookmarking) or just bad design.

HBI?

What is HBI? A quick search found the following unrelated and unhelpful information:

HBI Health and Biomedical Information
HBI Healthcare Building Ideas (magazine)
HBI Home Builders Institute
HBI Home Business Institute
HBI Horizontal Blanking Interval (television)
HBI Hot Beef Injection (band)
HBI Hot Briquetted Iron (plant or facility)
HBI Hubbard Broadcasting Inc.

Wikipedia: Page does not exist.

Re:HBI?

Humongously Bad Interface. That's the internal name for all new MS APIs.

Re:HBI?

I was assuming he meant Host Based Intrusion.

Microsoft and logs do not compute

I once had a 800MB plain-text logfile that I wanted to do a simple search and replace. I opened up the file in Word on a P4-2Ghz-2GB system and it took over two hours to complete roughly 50% of the task at hand. At this point I finally gave up because I was worried what was being done to my file and copied the file to an old PIII/450MHZ/512mb running linux and the task took about 2 seconds using a simple regex with sed.

Re:Microsoft and logs do not compute

Isn't that just you announcing your ignorant of which tools to use? Are you that kid in gym class that was always trying to put his shoes back on without untying them, rather than take the seconds to untie/re-tie he'd stomp himself around the locker room for minutes until they fit right. Oh and, how long would it take you to create and print a tri-fold pamphlet using sed? Perhaps you're the problem, not the app.

Swimming in acronym soup...

Could someone with more Microsoft Kool-Aid in their veins stick their fork in the acronym salad that is this article? ACL (Access Control Lists - which technically are a firewall), DoS (denial of service attacks) and IPS (intrusion protection services) I all know, but WTF are:

HBI?
GFS (is the G for "Ghost")?
NBI?
NLB?
ACE?

TIA :),
/tsg/

Re:Swimming in acronym soup...

Interesting, I thought I was the only one. Why is it that every time I read about Microsoft related technology it's always an acronym salad. Not even commonly used acronyms either, they use acronyms for their own way of calling technology xyz. It's almost like they do it on purpose ..

Re:Swimming in acronym soup...

GFS: Global Foundation Services. Microsoft's big internal network management thing. It's the people who keep the servers up and running for everything facing outward.

HBI: High Business Impact. Social Security numbers ,Passport accounts, etc.

NLB: Network Load Balancer.

AV: AntiVirus.
DoS: Denial of Service
IIS: Internet Information Services. 'httpd' for Windows.

Better response:

At this point we still don't use firewalls for MS.COM sites and don't have any plans on the books to put them in place. Here is the short answer as to why:

1. We run Linux.

What happened to Akamai Linux?

I vaguely recall MSFT had to outsource load balancing to Akamai which used Linux boxes to redistribute the incoming traffic at some point in the past. Looking at Netcraft.com, it shows some subdomains of microsoft.com resolved to Linux boxes before the year 2000. So it is able to get out of the sandbox now? Is that the main story?

Perhaps the only ones who can do it "right"

Let's set aside the natural urge to bash MS into oblivion. Let's (just for now) ignore conventional advice about network security and firewall use. Now, not only are these guys a Microsoft shop...they ARE Microsoft. MS claims their software is stable and secure. Perhaps it is -- when was the last time microsoft.com was taken down by malevolent hackers?

That said, with their closed source and closed-doors policy to revealing details about the inner workings of the OS, _Microsoft_ may be the only company that can successfully deploy a 100% Microsoft powered solution. How many registry changes, service daemon modifications, and other tweaks have been made to get their config running this way? The world may never know. It's probably impossible for the consumer world to ever have that level on knowledge about the Windows environment, and thus run it at peak security levels. For most consumers and businesses, a Linux OS with properly implemented firewalls is much more secure than an out-of-the-box Windows deployment and router ACLs.

akamai

don't forget the whole slough of Linux servers that they use through Akamai to handle the bandwidth;

it's one reason why why doing a lookup on Microsoft servers, it often shows that they are running Linux. It's also another reason why people point out that Linux is more scalable because even Microsoft can't eat it's own dogfood.

Ok...

Nice setup but what about root passwords?

Misleading Summary. Total Propaganda

1. The asshat highlights they use no firewall, and yet buried deeper in the article is this "Router ACLs are in place to block unnecessary ports" That's the functional equivalent of a firewall.

2. I get into discussions where tech guys spew traffic numbers and I'm never impressed. It creates issues if you want to actually do something with the data which I doubt they do much beyond running the usual marketing metrics. Until you actually shoot for 99.99 service uptime, you begin to comprehend the challenge it is (on any platform) the traffic itself is not the challenge.

3. I'm very interested in reading what their hardware budget is like. I get excellent performance out of Linux compared to server 2003 boxes on similar compaq dl380's.

Now there's a best practice

use of their yet unreleased Windows Server 2008 in a production environment.

Now there's a best practice that other corporations should follow - the use of test software in a production environment.

3 Free Tips

FTA

|In terms of how we protect the sites, we utilize (starting at the outside edge of the network and working in): ... Windows and IIS...rock solid and secure! www.microsoft.com is on Windows Server 2008/IIS7, MSDN/TechNet are migrating to Win2k8/IIS7, and update.microsoft.com is on Windows Server 2003/IIS6. ...

So there you have it. I think this is a good insight into how we run our own internet properties today. What do you think? Have you got any feedback for the boys over at our MSCOM Operations team?|

3 Free Tips, the rest I charge for:

1st don't advertise your networks security especial from the outside - in.
2nd don't believe your own propaganda on rock solid. There are too many issues in it to be rock solid.
3rd don't state your future migration plans on secure architectures to the public.

Cheers ;}

--- Just because you go hunting doesn't mean you have to shoot yourself in the foot ---

No Firewalls!

I have to say it sounds initially like they just stick the machines out there and let them fend for themselves. Then you read on and find that Microsoft can't live on the web without surrounding their servers by a ring of *nix devices providing Packet Filtering.

They don't trust even Win2k8 servers to be secure enough without the *nix safety blanket.

Back In The Days

I heard that Back In The days, Microsoft were using FreeBSD for their outward-facing servers, hacked-up to look exactly like Windows NT (for that was the product they were selling at the time).

perhaps microsoft will stamp one of these

"hacker tested" icons we see all over at various e-commerce sites then?

On the Subject of Firewalls and Microsoft.com

Every once in a while my Vista machine develops a little networking problem. I usually have to disable and re-enable the network card to bring it back. But, if I run Vista's network Diagnose & Repair first, a stupidity arises. It tries to ping www.microsoft.com, and when it fails, it complains. Why is it trying to ping www.microsoft.com? Www.microsoft.com does not reply to pings. Microsoft.com does (usually), but not www.microsoft.com. Www.microsoft.com resolves to lb1.www.ms.akadns.net, and IP addresses 207.46.19.190 and 207.46.192.254. A sample of the error message is bellow. [Window Title] Windows Network Diagnostics [Main Instruction] Cannot communicate with www.microsoft.com(207.46.192.254). [Content] Network diagnostics pinged the remote host but did not receive a response. [Reset the network adapter "Wireless Network Connection"] [Cancel] So, why on earth do they have the tool ping www.microsoft.com? Seems stupid to me.

He is basically clueless

The guy is clueless. All he wrote is that they use new version of IIS and Windows and nothing else. He does not have a clue on how it is all run. The OS they run is not important. I think the application stack is roughly equivalent to unix stuff. Despite the license fees which M$ doesn't have to pay.

1. For what I understand they don't handle data that needs some audit trail in transactions and so on so they don't need firewall. I don't see any logic in his statement.

2. 650GB/day (of what exactly?) may seem a lot but in fact a quite regular database cluster and a proper design would handle that easily if it is well scaled.

3. He is probably just quoting somebody else. Maybe he is right here but it is hard to judge with no knowledge on how exactly does this setup use? And what he means as firewall is another mystery for me.

4. He is stating that some form of NLB made by MS in their web server architecture is bad since it makes normal network design complex and expensive. Is that what he is stating?

5. This point also makes no sense to me. Of course application security is essential since it has nothing to do with firewall. A firewall merely passes or not the traffic based on simple, low-level protocol parameters. Firewall does not protect against application flaws. Application flaws occur at very different level. He is even clueless about OSI model...

The rest is just bullshit about how it is cool to use untested software in production. Actually it is very uncool.

Also this "knowledge" of his is useless. I would love to see some insights on such large setups from somebody who is not M$ and actually did research and testing on which platform to use. Like Google for example. :)

And also how does microsoft.com compares to google.com? Which is bigger in means of traffic/application load/databases and so on?

Microsoft shill doesn't know what a firewall is

News at 11.

Also, running AV software on a web server? What? I can't think of very many situations where that would be at all defensible.
The rest of the article reads like a marketing presentation. Very enterprise.

the article itself

since the link requires you to logon, here's a version of which I believe to be a copy of the article: http://www.networkmirror.com/EVCMz0uDTZ3L1XPV/blogs.technet.com/jeffa36/archive/2007/12/13/microsoft-com-what-s-the-story.aspx.html/ [networkmirror.com]

MS Xenix

Sure, that's all well and good for a public face but we all know they really use Xenix behind all that Windows dressing. ;-)

LoB

Bad link

The blog entry the story links to does not exist. Not Found: Forum Not Found The forum you requested does not exist.

This explains the crappy service

"the use of their yet unreleased Windows Server 2008 in a production environment."

Now how stupid is that? What sys admin would use an unreleased OS in a production environment?

That's like Rule No. 1, isn't it?

Hidden

The blog has been taken off public view, and only for those who have MS TechNET access. Before that, there were comments on lies & un explained abbreviations the dude used... /. word verification: bondage !!!

Some of the new stuff

I got to see some of the new stuff in Windows 2008 with one of the MS sales engineers, and I have to say, I'm impressed. Here's some of the stuff they did:
General:
This will be the last Windows Server that will have 32-bit installation available. With the popularity of x64 based Intel and AMD processors, and the proven reliability of WOW64, this shouldn't be a problem.
You may add/remove as many roles at a time, with a single reboot required after all the roles have been installed
You can bypass entering the product code on installation (Activation still requires the code though). Setup is no longer linear - you can pick and choose what you wish to configure.
Virtualization:
Virtualization has now become a feature of the OS, rather than a separate application installation. You can enable virtualization as a server role. When this happens, a thin layer acts as the interface between the virtual hosts and the hardware (marketing term: "Hypervisor"). The parent host OS then becomes a virtual image (that can't be moved). All hosts are treated as equals.
Virtualization requires the 64-bit edition of Server 2008 installed.
Virtual machines can now have memory spaces > 4 GB and have multiple cores
Virtual machines can run any Windows and some Linux variants are now supported (most likely all will run; MS will actually field support calls for the supported Linux variants).
Event Log
The event log is so much better that I can't begin to explain how much better is it. You truly have to see it. Here's some of the features:
Events displayed within each subsystems management screens. Ex: if I were to open IIS management, I would see a default screen with all the events that were generated by IIS, and none that were generated by other systems.
Events from all eventlogs (Application, Security, System, etc) can be displayed in one window
You are able to see events categorized by event severity, and grouped by time frame (ex: 1 critical event in the last hour, 3 in the last day, x in the last week).
You are able to push events to a central server from multiple server, or you can pull events from other servers to one (subscription)
You are able to execute applications or send emails when an event is fired. You set up criteria for that to happen (event ID, severity, text in body/subject, etc).
Management
The Computer Management MMC console has been replaced by the Server Management console. The Server Management console is automatically populated with links to the management windows for each installed role, thus making it the de-facto configuration window.
PowerShell is a new command line interface. It is a hybrid console/scripting environment, created to aid in systems management. You can manage either the local server or remote servers from it.
New Server 2008 Core Installation Option
Server core is an optional way to implement Windows 2008. It removes the GUI portion of the OS as well as a number of other features, thus reducing the attack surface of the OS.
Core is not a separate product; the Standard, Enterprise, and Datacenter editions can all be installed in Core mode
Managed with remote tools and command prompt (cmd)
5 available server roles
Included:
o DNS
o DHCP
o File sharing
o AD
o WSV - windows server virtualization
o Limited IIS - static content only
o Task manager
Not included:
o No GUI

URL Not Working...

Here, fixed that for you: http://www.networkmirror.com/EVCMz0uDTZ3L1XPV/blogs.technet.com/jeffa36/archive/2007/12/13/microsoft-com-what-s-the-story.aspx.html [networkmirror.com] Enjoy! :)

Take it easy

If you use the definition given by Wikipedia of a firewall http://en.wikipedia.org/wiki/Firewall [wikipedia.org], you will notice that the first generation of firewalls were in fact packet filters. However, as with many words or phrases, definitions change over time. The definition commonly associated with a modern firewall is something more than a simple packet filter like an application layer firewall or stateful filter. For us Linux, Cisco or other old school IT guys, we still refer to a packet filter as a firewall. This person obviously has a more modern, Microsoft, way of thinking of a firewall. That does not give us the right to belittle him and say he does not know what he is talking about. They have a certain level of security, whether you want to call it a firewall or not is your choice. To be perfectly honest, a packet filter is not much with security these days. Most attacks are going to be directed to a certain port to exploit specific software vulnerabilities and these are the attacks that a packet filter cannot handle. A bigger security risk is the fact that they are using unproven software to run their production environment. I personally wouldn't use a new windows OS until, at least, service pack 1 in a production environment. It is always best to wait for software to be proven before it is allowed in a production environment (see Debian GNU/Linux).

Asked to log in before reading a blog? What?

Why am I being asked to log in before I read a fucking blog? I've tried both Firefox and Safari now, and both ask me to log in. Even going to the root blog URL redirects me to a login page.

Link broken?

When I go to that link with the bugmenot login, I get:
Not Found: Forum Not Found
The forum you requested does not exist.

Was the article deleted?

Isn't it ironic?

We are currently unable to serve your request

We apologize, but an error occurred and your request could not be completed.

This error has been logged. If you have additional information that you believe may have caused this error please report the problem here.

Looks like that 650 GB of logs is going to be bigger today...

hum ...

We are currently unable to serve your request

We apologize, but an error occurred and your request could not be completed.

This error has been logged. If you have additional information that you believe may have caused this error please report the problem here.

this is what I get

when I try to go to their site:

"We are currently unable to serve your request

We apologize, but an error occurred and your request could not be completed.

This error has been logged. If you have additional information that you believe may have caused this error please report the problem here.
"

I think that gives a good demonstration of how they run their site...

We are currently unable to serve your request

We are currently unable to serve your request

Slashdotted. Oh, the many levels of delicious irony...

We are currently unable to serve your request

We apologize, but an error occurred and your request could not be completed.
This error has been logged. If you have additional information that you believe may have caused this error please report the problem here.

The above is what I get when I try to RTFA. I guess that tells me all I need to know!

Post deleted

The article seems to have been deleted -- everything else works, except it can't find that article. Google Cache works, though. http://209.85.173.104/search?q=cache:cZIWXV4A-GIJ:blogs.technet.com/jeffa36/archive/2007/12/13/microsoft-com-what-s-the-story.aspx+http://blogs.technet.com/jeffa36/archive/2007/12/13/microsoft-com-what-s-the-story.aspx&hl=en&ct=clnk&cd=1&gl=nz&client=firefox-a [209.85.173.104]

Re:A router can be a firewall too

HBI means "High Business Impact [google.com]" in Microsoftese. There are also MBI, meaning "Moderate Business Impact", and LBI; "Low Business Impact".

Re:They do use firewall

Maybe he meant that the building itself has no walls to protect it from a fire. Maybe their server room is in a gazebo in a park somewhere.

Re:They do use firewall

No, dufus. A true firewall inspects individual packets.

Re:Router ACL= Firewall

SUREURCORRECT!

2. Router ACLs are in place to block unnecessary ports

Right-o ! Shows what a brainwashed, single-minded dim he is. Doesn't say "(Microsoft) Firewall v.0.38.2a" on the shrink-wrapped package; and voilà, isn't (a firewall). That's how they keep the masses unwashed and in admiration. (But I digress.)

Actually, the whole thing is a disgrace, but what to expect ... !?

2. We have ~650GB/day of IIS logs [...] Just IIS logs are a challenge without trying to parse another ~650GB of firewall logs.

Why is an IIS log size just as large as a firewall log ? Makes me wonder, if he thinks they were the same ??
650GB of what ? ASCII text or gzip ?

3. 5+ years ago, there wasn't a firewall solution that would scale to our needs and this forced us to focus on network, host, and application security.

I'd never would want their stuff for free even. Because the use of the word 'forced' is absolutely wrong. Program security is the alpha and omega of security; and anyone who wants to have his software taken seriously would look into exactly these. Not into firewalls.

5. Application security is critical since a firewall is likely going to allow traffic on the correct port and protocol through to the web servers so IIS/ASP.NET/Applications must deal with these requests gracefully.

This is so right, see above. But the mentality implies he is unaware of the fact that predictable and graceful behaviour is what we want in the applications in the first place.

6. We do run AV on our servers when we can. At times product adoption means we don't install it, but we do normally run AV.

Makes one wonder what this is supposed to tell us. At times they don't get an AV running on their own boxen ? Can someone point out to me, which logic underpins non-usage of AV for 'product adoption' ? Like, on those boxen containing Vista ?

Re:No filewall?

hey what verion of nmap are you running?

Re:A router can be a firewall too

2. [...] Just IIS logs are a challenge without trying to parse another ~650GB of firewall logs.

That's a non-argument. I use iptables without the LOG target; why would i want to log packets before dropping them? This would make no sense to me. If i want a NIDS, i'll install a NIDS.

IIS is a web server, thus those are web server logs, which can be parsed to get statistics about page views, errors, etc...

Re:nothing better to say?

Actually, all that the Microsoft infrastructure says is that they have put together a pretty good & scaleable server farm/cluster solution, nothing more. The same can be done using Linux servers and clusters in theory - has it been done with Linux on the same scale as Microsoft.com? I don't know.

And if you're referring to Linux versioning, please remember that with OSS products there is no remit to get a "finished" product into a box onto the shelves - just because it happens to be "Random Linux App v0.3" does not mean it is "not quite done".

Re:It Blows

Where does it say 'error logs'? I read 'IIS logs'.

Re:Bill Gates Behind a Curtain

The Bill of Oz?

But it's a setup
until you're fed up

Re:It Blows

Allow me:

• We are currently unable to serve your request
  
  We apologize, but an error occurred and your request could not be completed.
  
  This error has been logged. If you have additional information that you believe may have caused this error please report the problem here.

How appropriate seen articles subject.
Hah.