Akamai: How They Fought Recent DDoS Attacks

yootje writes "Infoworld is running an interesting article about Akamai and the DDoS attack that hit the network of Akamai Tuesday. According to this article one of the defenses of Akamai is the big diversity of their hardware: 'We deliberately use different operating systems, different name server implementations, different kinds of routers, different kinds of switches, different kinds of CPUs, and especially, different operational procedures.' So says Paul Vixie, architect of BIND and president of the ITC." Yootje points to another article on this subject as well, this one at Internetnews.com. Update: 07/07 19:38 GMT by T : Note that Vixie's quote here is actually presented out of context; he was commenting by way of contrast on the diversity of the root DNS servers, not Akamai's content-serving system.

Trade-Off

[cynic10508]

The diversity of hardware and software may be an IT nightmare but I think this shows how effective it really is. Now all we need is a concise cost/benefit analysis.

security by obscurity..

[klang]

nobody knows what they run, so nobody can make a decent attack ..

intentional or not

[cjwl]

I have to wonder if the diversity of systems was an intentional choice of theirs way back to face these kinds of attacks or if it just grew that way from rapid growth and having their systems spread all over.

They survived the attack and "Oh yea, we MEANT for it to happen that way".

I think it's spin.

Submitters and Editors, RTFA!

[adavies42]

The quote on diversity is by Vixie wrt the roots servers--it's a criticism of Akamai! Jesus H. Christ, it's in the first paragraph!

Re:Trade-Off

[Ignignot]

Allow me to perform a concise analysis for you. Hmm... the benefits are that DDoS's have some trouble knocking you offline. What are the costs? Much higher IT costs. Also, the total number of holes in your security will be higher. Just keeping track of all windows security fixes is hard. Imagine doing that for windows, solaris, linux, osx, and bsd. On 100 different hardware setups. Some things are going to go unpatched. You're giving hackers / crackers more opportunities, not more problems.

Re:security by obscurity..

[stratjakt]

Sort of. You can know what they run, you can know you can exploit server A because it has a known vulnerability.

But servers B, C, D, E, F, G, etc are immune to your attacks on server A. To take down the root servers, you'd need to simultaneosly come up with 12 different exploits to knock each one of them out. Which makes it 12 times more difficult.

It's more proof of what I've always said, there is no "perfectly secure" OS in existence.

This is an ad!

[isaac]

This article has nothing to do with Akamai, other than pointing out that Akamai DNS is vulnerable to DOS.

Most of this "article" is a puff-piece (or paid advert) for one "CloudShield Technologies," pimping their (vaporware) "server for applications that do deep packet processing at gigabit-per-second rates."

-Isaac

Re:security by obscurity..

[qtone42]

Oh, yeah. We got Death Star.

Re:Trade-Off

[Pharmboy]

Even with our little network (2 T1s, several servers) we do the same thing. Different OS versions, Bind builds, even Apache implimentations. NS1 is dedicated on a slow but extremely robust dual cpu box, all other boxes have a primary task and act as a back up for other tasks. At this small level, its not THAT hard to do, although it takes some preplanning and maintenance. Even the outbound linux router has an offline spare with a different version of Linux and completely different firewall/NAT configuration in case the first gets taken down.

IMHO, when it comes to providing IT services, if you are not paranoid, you are crazy.

Re:Trade-Off

[Tony-A]

Now all we need is a concise cost/benefit analysis.

Life versus death?

What you want out of backups and backup systems isn't so much that they are as good as or better than the primary systems, but that they are as independent as possible. Backing up OpenBSD to Windows 95 is not as stupid as it looks.

Re:Sys admins

[ron_ivi]

different operating systesm ... Wow, your sys admins and help desk must LOVE supporting that!

I know you were trying to be sarcastic, but I bet that they indeed do prefer things this way.

When the pager goes off at 3AM that there's a suspected new worm attacking your dos-based systems, it's nice to simply turn them off and let the other systems handle the load until morning when you can investigate the problem at your leisure.

Authors should try readin the article

[rgmoore]

According to this article one of the defenses of Akamai is the big diversity of their hardware: 'We deliberately use different operating systems, different name server implementations, different kinds of routers, different kinds of switches, different kinds of CPUs, and especially, different operational procedures.' So says Paul Vixie, architect of BIND and president of the ITC.

Actually, according to the article the diversity approach is part of what's used to defend the DNS root servers, not Akamai. Vixie specifically mentions that this approach is not practical for an ordinary content provider like Akamai because, 'the cost would "drive their accountants crazy."' I'm dubious about just how helpful diversity would be against a DDoS attack in the first place. Diversity won't solve the problem of requests coming in faster than they can be processed.

Re:Quote misattributed

[NekoXP]

Having your sysadmins LEARNING how to use new architectures, procedures and so on costs money - because their time is on salary, you pay for that learning process, their lack of knowledge in the beginning adding time to solving problems, and bringing in help costs more because you'd prefer they'd have that broad experience already.

Remember.. [insert product here] is free if your time is worthless.

Neko

Re:Trade-Off

[bastardadmin]

If you are Akamai, your uptime isn't everything, it is the only thing.

In their case maintaining a hybrid infrastructure makes perfect sense.
Remote exploit in IOS? No problem, the Juniper/Extreme/Linux/OpenBSD router in failover config takes over while patching goes on.

And if you are maintaining a massive hybrid infrastructure like that you will likely have the people and processes to handle security issues/patches.

Security through obscurity..

[CokoBWare]

A valid tactic... it mitigates the problems with a unified vendor, but it costs lots more...

Re:intentional or not

[Radon Knight]

I think it's spin.

Maybe so, but there's a kernal of truth there. Diversity in biological systems produces robustness. If you have a rich genetic code in a species, you're more likely to have a subset of the population that will survive a new virus, disease, etc. Given the complexity of networked computer systems, is it really that surprising that we're finding certain survival techniques which work well in nature work well when applied in alternative environments?

That idea's not new, and it's not well-defined. However, I would certainly like to see it made more precise and analyzed so that we can see just what, really, lies at the bottom of that otherwise vague analogy.

Gee-Wiz hardware will never win.

[twitter]

[description of magnificent gateway] For now the attackers are winning the arms race. The technology we'll need to monitor, react, and adapt in real time has yet to evolve, but it's headed in that direction.

I wish the net was headed in the right direction, but it's not. No single site or company will ever "win". The resilience of the web lies in it's redundancy and distribution. What I see is continued centralization and creation of points of failure. As "Broadband" internet access is more monopolized and treated as a platform for mindless browsing, and smaller ISPs are destroyed, the net is being squeezed into fewer and fewer hands. This invites attacks that can not be protected against. The real solution is to let everyone run everthing they want. That's the only way to route around damage.

Attacking Akamai with a DDoS...

[Mr. Neutron]

...is like trying to wipe out swarm of gnats with a shotgun.

Re:Trade-Off

[freqres]

At least now in federal courts, any monetary damages used to determine sentencing must now be presented and supposedly proven in front of a jury during the trial. Much better than the federal prosecution creating huge dollar sum damages during the sentencing phase with little burden of proof. I guess the Supreme Court gets something right every so often (much like the blind squirrel and his nuts I guess).

Re:Trade-Off

[Anonymous Coward]

So, in this case, not only did the submitter not read the article, but neither did the editors. I actually read the article and it was blatanly clear the the whole heterogeneous argument was *not* in reference to Akamai.

I just have one question: what exactly do the slashdot editors do? I thought they were there to screen incoming submissions. But obviously they don't. Basically, if that's their only job, they suck at it.

Re:Trade-Off

[johnnyb]

However, you are preventing your entire infrastructure to being nailed by a single exploit. With a monoculture, a single flaw exploited by a worm can destroy pretty much everything. With a mixed setup, although you have more possible entrances, each one allows a lot less damage.

If I have 1,000 troops, if I keep them all in the same fort, they will be a formidable force, unless I find the right weapon (like a nuke). If I keep them in 10 different forts spready throughout the country, although each one of them is more vulnerable individually, I have eliminated the possibility of everything being wiped out in a single blow.

Re:Quote misattributed

[2names]

The workplace is not a classroom, nor should it be treated as such.

If you have not realized that every place is a classroom, then, my friend, you have not learned a single thing.

Re:Sys admins

[SpaceCadetTrav]

Actually, at 3AM they're probably still awake, trying to figure out how to get all these different systems to behave exactly the same under normal operating conditions.

Re:Attacking Akamai with a DDoS...

[Anonymous Coward]

Nobody said a shotgun wouldn't work at all, it just wouldn't work very effectively.

Yootje Points?

[Telepathetic Man]

What the heck are those? Are they like bad karma points for articles that have overlapping information with other articles?

By the way, which one of the articles is it that says Akamai did anything right to fight attacks?

Article isn't about the DDOS

[np_bernstein]

'It's about CloudShield Technologies ... recently announced CS-2000', and nothing but a fluff peice meant to sell some hardware. Sure, Akami's DDOS is discussed ("DDOSs are ba-ad, mmkay."), but then it just goes on to talk about the CS-2000.

nobody read anything

[Anonymous Coward]

not only did the submitter not rtfa

the editors did not rtfa

and after the first five posts pointing this out, it was obvious that nobody was reading the responses either.

nobody was reading anything, and now we have a 1000 responses saying the same thing, it wasn't akamai, it was the root servers, blah blah blah.

Re:Sys admins

[Anonymous Coward]

different operating systesm ... Wow, your sys admins and help desk must LOVE supporting that!

I know you were trying to be sarcastic, but I bet that they indeed do prefer things this way.

When the pager goes off at 3AM that there's a suspected new worm attacking your dos-based systems, it's nice to simply turn them off and let the other systems handle the load until morning when you can investigate the problem at your leisure.

Actually, they love it because when the pager goes off at 3AM, they know the backups are able to take over so they can work on what is causing the problem, and have everything back up and running by 8:00am when the boss walks through the door. Otherwise they end up scrambling at 3:00am to get something, _anything_, up and running so your critical services can be restored BEFOR working on the problem at hand.

Re:Sys admins

[LookSharp]

Can I ask an obvious question here?

Who the atech-ee-double-hockey-sticks runs "dos-based" systems anymore? I thought Microsoft abandoned the technology starting in 1995, and I personally submitted the "official end of life for DOS support" article to Slashdot several years ago.

We run heterogenious systems and support them because they provide different benefits and features for our many needs. Sometimes Windows OS servers actually are cheaper, more stable, and easier to support than their Unix counterparts. Sometimes not.

For instance, we have WebSphere running on Solaris and AIX as an app server platform, and it is great for high volume and failover. But we spend far more time (proportionally) troubleshooting that technology (and the hundred or so servers that run it) than the .NET application servers running on Windows 2000. As an app environment .NET is stable and actually quite fast, and run on much less expensive equipment. However there are only four of them and failover between boxes is sketchy, so on the rare occasion that there is a non-code related outage, it takes longer to get the environment back up to spec.

Just my anecdotal experience.

Re:Read the fucking article before submitting it

[yootje]

Dude, calm down. I'm sorry, I admit I wanted to have it fast on Slashdot, but not for my ego, but I like it to have it on Slashdot quick. You are talking to real persons, it was a mistake. Come on, it's not like your life depends on Slashdot.

Re:Fuck

[Anonymous Coward]

Also please make sure it's not a paid ad for some ByMeNow-5000 product rather than an actual article.

Re:Diversity Doesn't Refer to Akamai at All

[Zeinfeld]

AFAIK, all of the text that the quote from the submitter is regarding not Akamai, but BIND in criticism of Akamai. He's saying that they would have performed better had they used a more diversified network

Paul should shut up about this topic. Companies should not go commenting about attacks made against their competitors - period.

His statement about the root servers is way off base. Only four of the 13 servers stayed up and the software running on them did not affect the outcome in any way. Most of the servers that went down were running a version of BIND as were two of the servers that stayed up. The other two roots were running ATLAS which is the ultimate in closed source proprietary systems, nobody outside VeriSign has seen the executable, let alone the source code.

I don't see how anyone could draw any conclusions either way on the basis of this sample. The distinguishing feature was the bandwidth available to the systems, not the software they run.

Paul should think more and speak to journalists less.