Akamai: How They Fought Recent DDoS Attacks 231
yootje writes "Infoworld is running an interesting article about Akamai and the DDoS attack that hit the network of Akamai Tuesday. According to this article one of the defenses of Akamai is the big diversity of their hardware: 'We deliberately use different operating systems, different name server implementations, different kinds of routers, different kinds of switches, different kinds of CPUs, and especially, different operational procedures.' So says Paul Vixie, architect of BIND and president of the ITC." Yootje points to another article on this subject as well, this one at Internetnews.com. Update: 07/07 19:38 GMT by T : Note that Vixie's quote here is actually presented out of context; he was commenting by way of contrast on the diversity of the root DNS servers, not Akamai's content-serving system.
Wow (Score:5, Funny)
That's shocking!
afternet (Score:2)
they are totally hosed right now due to a huge ddos.
see http://www.afternet.org/ for all the details
sucks
Trade-Off (Score:5, Insightful)
Re:Trade-Off (Score:4, Insightful)
Re:Trade-Off (Score:2, Interesting)
Re:Trade-Off (Score:5, Insightful)
In their case maintaining a hybrid infrastructure makes perfect sense.
Remote exploit in IOS? No problem, the Juniper/Extreme/Linux/OpenBSD router in failover config takes over while patching goes on.
And if you are maintaining a massive hybrid infrastructure like that you will likely have the people and processes to handle security issues/patches.
Diversity Doesn't Refer to Akamai at All (Score:5, Informative)
Correct me if I'm wrong.
Re:Diversity Doesn't Refer to Akamai at All (Score:5, Insightful)
Paul should shut up about this topic. Companies should not go commenting about attacks made against their competitors - period.
His statement about the root servers is way off base. Only four of the 13 servers stayed up and the software running on them did not affect the outcome in any way. Most of the servers that went down were running a version of BIND as were two of the servers that stayed up. The other two roots were running ATLAS which is the ultimate in closed source proprietary systems, nobody outside VeriSign has seen the executable, let alone the source code.
I don't see how anyone could draw any conclusions either way on the basis of this sample. The distinguishing feature was the bandwidth available to the systems, not the software they run.
Paul should think more and speak to journalists less.
Re:Trade-Off (Score:5, Insightful)
If I have 1,000 troops, if I keep them all in the same fort, they will be a formidable force, unless I find the right weapon (like a nuke). If I keep them in 10 different forts spready throughout the country, although each one of them is more vulnerable individually, I have eliminated the possibility of everything being wiped out in a single blow.
Re:Trade-Off (Score:2, Interesting)
Its Slow death.
Re:Trade-Off (Score:5, Insightful)
IMHO, when it comes to providing IT services, if you are not paranoid, you are crazy.
Re:Trade-Off (Score:4, Insightful)
Life versus death?
What you want out of backups and backup systems isn't so much that they are as good as or better than the primary systems, but that they are as independent as possible. Backing up OpenBSD to Windows 95 is not as stupid as it looks.
Obligatory Simpsons Reference (Score:2)
lol, you are correct! One of our backup solutions is having a win98 box with ActivePerl installed go grab a copy of the datafiles every night. In the event of data corruption, THAT is usually the copy I restore from, purely because it is fast to restore from and highly reliable. (Yes, reliable. It only crashes when you are doing something, so it gets rebooted often enough
Not sure why, but that reminded me of the Simpson's episode where
Re:Obligatory Simpsons Reference (Score:2, Funny)
(Yes, reliable. It only crashes when you are doing something, so it gets rebooted often enough ;)
That's it. My reading comprehension is gone. I'm going to bed now and hopefully not dream of anything remotely related to this Daliesque image.
Re:Trade-Off - TCO (Score:3, Funny)
Just ask MS, they will tell you.
Re:Trade-Off (Score:5, Informative)
Mod this whole story down "-1 incorrect".
Re:Trade-Off (Score:4, Insightful)
I just have one question: what exactly do the slashdot editors do? I thought they were there to screen incoming submissions. But obviously they don't. Basically, if that's their only job, they suck at it.
Re:Trade-Off (Score:3, Interesting)
Basically, it works like this
Also, man hours get factored in, sometimes two or three times over, including the man hours that were used to create the product in the first place, as well as to re-create the product again.
It's all very stupid, and nobody believes a word of it except the courts.
Cause
Re:Trade-Off (Score:2, Insightful)
Sys admins (Score:5, Funny)
Wow, your sys admins and help desk must LOVE supporting that!
Re:Sys admins (Score:5, Insightful)
I know you were trying to be sarcastic, but I bet that they indeed do prefer things this way.
When the pager goes off at 3AM that there's a suspected new worm attacking your dos-based systems, it's nice to simply turn them off and let the other systems handle the load until morning when you can investigate the problem at your leisure.
Re:Sys admins (Score:5, Insightful)
Who the atech-ee-double-hockey-sticks runs "dos-based" systems anymore? I thought Microsoft abandoned the technology starting in 1995, and I personally submitted the "official end of life for DOS support" article to Slashdot several years ago.
We run heterogenious systems and support them because they provide different benefits and features for our many needs. Sometimes Windows OS servers actually are cheaper, more stable, and easier to support than their Unix counterparts. Sometimes not.
For instance, we have WebSphere running on Solaris and AIX as an app server platform, and it is great for high volume and failover. But we spend far more time (proportionally) troubleshooting that technology (and the hundred or so servers that run it) than the
Just my anecdotal experience.
DOS (Score:2)
Wow... (Score:5, Funny)
Re:Wow... (Score:2, Funny)
Jeremy
Re:Wow... (Score:2, Funny)
WRONG! (Score:5, Informative)
Re:WRONG! (Score:5, Informative)
security by obscurity.. (Score:4, Insightful)
Re:security by obscurity.. (Score:5, Insightful)
But servers B, C, D, E, F, G, etc are immune to your attacks on server A. To take down the root servers, you'd need to simultaneosly come up with 12 different exploits to knock each one of them out. Which makes it 12 times more difficult.
It's more proof of what I've always said, there is no "perfectly secure" OS in existence.
or... (Score:5, Funny)
that's been proven to be an effective, system independent DoS attack (even if the attack was unintentional or brought about by the owner)
Re:security by obscurity.. (Score:3, Insightful)
Re:security by obscurity.. (Score:2, Informative)
nobody knows what they run, so nobody can make a decent attack ..
Well, Kerkoff (sic) said in his principles of security to make the paranoid assumption that attackers will always be able to know what you have and/or how it works. So he says security only by obscurity isn't security at all. Kind of like the ostrich sticking its head in the sand and hoping the lion doesn't see it.
Security through Stupidity (Score:2)
Re:Security through Stupidity (Score:4, Funny)
ver^M
MS DOS 6.22
"wtf?"
Re:security by obscurity.. (Score:2)
Quote misattributed (Score:2, Informative)
Re:Quote misattributed (Score:5, Informative)
Exactly. And Vixie goes on to say that Akamai can't do that because "the cost would 'drive their accountants crazy.'".
But I'm not sure having diverse bits of gear is such a huge cost. Wouldn't it instead be a way for sysadmins to broaden their experience and learn more about which tools are best for which jobs?
Re:Quote misattributed (Score:2, Insightful)
Remember.. [insert product here] is free if your time is worthless.
Neko
Re:Quote misattributed (Score:3, Interesting)
Accounting has to be able to cost-justify purchases, otherwise they would be open to easy abuse. Therefore, you have to show that they need sufficient load on the servers to justify the ex
Re:Quote misattributed (Score:2)
Yes.
> the transaction server is down
> better than 10-20%
I'm not sure that necessarily follows from having a diverse collection of gear.
> The workplace is not a classroom,
> nor should it be treated as such.
Of course it is, and it should be. Usually it's referred to as "on the job training".
> you learn as you go,
Right on.
Re:Quote misattributed (Score:5, Insightful)
If you have not realized that every place is a classroom, then, my friend, you have not learned a single thing.
Re:Quote misattributed (Score:3)
Re:Quote misattributed (Score:3, Funny)
The editors don't read the articles, so why should the submitters be subjected to the same burden?
Re:Quote misattributed (Score:3, Funny)
Re:Quote misattributed (Score:2)
No, it's about "one" particular root nameserver [root-servers.org], F-root [isc.org], which is the root ISC [isc.org] operate. It's one IPv4 address, but actually a whole bunch of machines located across the world [isc.org].
Lack of diversity (Score:2, Redundant)
Re:Lack of diversity (Score:4, Interesting)
Re:Lack of diversity (Score:2)
Re:Lack of diversity (Score:2)
intentional or not (Score:4, Insightful)
They survived the attack and "Oh yea, we MEANT for it to happen that way".
I think it's spin.
Re:intentional or not (Score:3, Insightful)
Maybe so, but there's a kernal of truth there. Diversity in biological systems produces robustness. If you have a rich genetic code in a species, you're more likely to have a subset of the population that will survive a new virus, disease, etc. Given the complexity of networked computer systems, is it really that surprising that we're finding certain survival techniques which work well in nature work well when applied in alternative environments?
That idea's not new, and it's not well-
Re:intentional or not (Score:2)
They never mention percentage of users impacted (Score:5, Interesting)
Theo only statistic they ofer is the percentage of customers that were impacted. To me this hints of trying to play down the severity of the situation. When only 2 percent of your customers comprise (following is is a made up statistic since they didn't give me one) 80 percent of your traffic, you're lying by omission by only giving customer statistics.
Re:They never mention percentage of users impacted (Score:2)
Re:They never mention percentage of users impacted (Score:2)
There are lies, damn lies, and statistics.
Believe me, you can take any set of numbers and put whatever spin on them you want; one small fact can not paint a picture by itself. The real question is how accountable are you to the people you're quoting statistics to... in this case, the audience of the message is their current customerbase, in hopes of retaining them, and potential customers, in hopes of not scaring them away.
In both cases you ha
Re:They never mention percentage of users impacted (Score:2)
I would bet that anyone who has services from akamai is fairly high traffic, otherwise what is the point. Akamai's not cheap, and people wouldn't be using Akamai if there wasn't a need!
That said, I doubt 2% of their customers would be responsible for 80% of the traffic. . .
It's all relative (Score:3, Informative)
Akamai is, at best, being disingenous when they say only 2 percent of their customers were affected by the outage. Maybe 2 percent of their customers, but how many of their customers customers were affected?
2 percent may not sound like much on the surface, but if that percentage includes companies like Microsoft, MSNBC, Amazon, Yahoo, CNN, Lycos and other big-shot content providers then the relative number of "customers" affected by the outage is a lot more notable.
The submitter is WRONG. (Score:3, Informative)
Diversity of hardware makes ROOT DNS SERVERS more defensible. Akamai is NOT diverse, and they do not want to be.
Submitters and Editors, RTFA! (Score:4, Insightful)
This is an ad! (Score:5, Insightful)
Most of this "article" is a puff-piece (or paid advert) for one "CloudShield Technologies," pimping their (vaporware) "server for applications that do deep packet processing at gigabit-per-second rates."
-Isaac
Re:This is an ad! (Score:2)
What the heck is "deep" packet processing anyway?
"deep" packet processing (Score:2)
Packets are composed in layers. The lower ones have to do with transmission over the network. The higher ones have to do with the interpretation of the packet (like which application session etc it belongs to). And of course you have the payload, the data being sent, like the letter in the envelope.
Mu!
Authors should try readin the article (Score:5, Insightful)
Actually, according to the article the diversity approach is part of what's used to defend the DNS root servers, not Akamai. Vixie specifically mentions that this approach is not practical for an ordinary content provider like Akamai because, 'the cost would "drive their accountants crazy."' I'm dubious about just how helpful diversity would be against a DDoS attack in the first place. Diversity won't solve the problem of requests coming in faster than they can be processed.
Uh, poster got it wrong (Score:2)
Erm, I think the poster made a mistake here. This diversity is attributed to the 13 root servers. Akamai's services do not employ such techniques due to the unsupportable cost. Based on the problems we saw during the DDoS, I can't say Akamai had much to offer in its arsenal.
Or am I the one who misread?
Different OS's? (Score:2, Funny)
So what they're saying is... (Score:4, Funny)
It sounds like a recipe for success!
Security through obscurity.. (Score:3, Insightful)
Re:Security through obscurity.. (Score:2)
Obscurity is hiding your dinosaur, hoping the meteorite won't see it.
Gee-Wiz hardware will never win. (Score:5, Insightful)
I wish the net was headed in the right direction, but it's not. No single site or company will ever "win". The resilience of the web lies in it's redundancy and distribution. What I see is continued centralization and creation of points of failure. As "Broadband" internet access is more monopolized and treated as a platform for mindless browsing, and smaller ISPs are destroyed, the net is being squeezed into fewer and fewer hands. This invites attacks that can not be protected against. The real solution is to let everyone run everthing they want. That's the only way to route around damage.
Attacking Akamai with a DDoS... (Score:5, Insightful)
Re:Attacking Akamai with a DDoS... (Score:4, Interesting)
Good old PR spin - nothing like it... (Score:5, Funny)
CTO: "Actually, sir, the real question is why did we lose less than half of our service. The answer is that I've, uh, been strategically using different systems and components throughout the enterprise on purpose to prevent drastic losses. No one else could have even kept 10% of their machines up under that DDOS."
Boss: "I knew I could count on you for the right PR spin job. Go back and think up some other good excuses."
-Adam
Ummm.. (Score:5, Interesting)
In the case of the Akamai incident, the vulnerable service was DNS. Paul Vixie, architect of BIND (Berkeley Internet Name Domain) and president of the Internet Systems Consortium, charged that Akamai's proprietary approach to DNS makes it a single point of failure. He added that the 13 DNS root servers, which weathered a vicious DDoS attack in 2002, are even more defensible today than they were back then. The root servers are resilient, Vixie said, because their operators embrace diversity. "We deliberately use different operating systems, different name server implementations, different kinds of routers, different kinds of switches, different kinds of CPUs, and especially, different operational procedures," Vixie told Internetnews.com.
He's not talking about how great Akamai is. He's talking about how great everyone else is.
On another note: What the heck does this story have to do with Akamai operators fighting DDoS attacks? They more than likely sat with their thumbs up their rears contemplating how having such a structured and inflexible DNS system could possibly be in err.
Interesting... (Score:2)
I was way off... (Score:3, Funny)
'We deliberately use different operating systems, different name server implementations, different kinds of routers, different kinds of switches, different kinds of CPUs, and especially, different operational procedures.'
RTFA first, please... (Score:3, Informative)
Oooops (Score:3, Informative)
Yootje Points? (Score:2, Insightful)
By the way, which one of the articles is it that says Akamai did anything right to fight attacks?
extra secure systems (Score:2, Funny)
They called me crazy for using Windows 95, 98, 2000, CE and ME . . . I'm invincible! Bwahahaha!
Article isn't about the DDOS (Score:3, Insightful)
nobody read anything (Score:4, Insightful)
the editors did not rtfa
and after the first five posts pointing this out, it was obvious that nobody was reading the responses either.
nobody was reading anything, and now we have a 1000 responses saying the same thing, it wasn't akamai, it was the root servers, blah blah blah.
Fuck (Score:5, Funny)
Re:Fuck (Score:3, Insightful)
Same cause as recent big electrical blackouts (Score:2, Funny)
(BOFH types RETURN, followed by)
"Oh Shit!"
Article is an ad for Vixie and his companies... (Score:3, Informative)
Second, he is being disingenuous about his comments about patents, his company owns at least one patent related to the Verisign "Site Finder" service methodology. Nominum Patent [uspto.gov] I didn't see any statements by him disparaging his company when they applied for that patent. So it isn't that he doesn't like patents, it is that he doesn't like that Akamai is making money doing third party DNS without paying him money or homage. Note: His commercial, for profit dns server software company has a white paper enumerating the scalability and other problems with BIND, and they use an architecture more similar to DJBDNS than to BIND 9 - separate auth and resolving dns server packages, most modern dns server software uses this architecture to reduce code complexity and improve security and performance.
Third, if he wanted to be the pillar of dns server software that he supposedly is, he could have sent a few goons from Nominum over to Akamai and set up some boxes with his commercial, for profit, "scalable" dns server software and Akamai would have been able to see if his software was able to stand up to the ddos attack better than what they have. If it did, he probably could have gotten a sweet, lucrative contract out of it and been a hero for helping thwart the attack, rather than a hypocritical, self serving competitor hiding behind Open Source to appear credible.
Fourth, Akamai is a single point of failure because that is what they do - offload dns and content load from the biggest companies on the net life MS, google and ebay. No, I don't work there, but I would venture a guess that they carry more traffic than (maybe) any other company. So I am sure it is easy to armchair quarterback and say they should do this and that, but when the attacks are probably at 10's or 100's of GiB/s I am not sure what I would do.
Nominum is also involved in RFID stuff, so I will be interested to see what happens with him and his companies as that ramps up. And who knows what deals have already been made - "the future of DNS is right."
Some DNS software links:
nsd - high performance, uses BIND style files and authoritative only [nlnetlabs.nl]
They have an interesting testing procedure where they run nsd and BIND, have them build responses to the same queries and then analyze any differences: diff analysis [nlnetlabs.nl]
maradns [maradns.org]
Powerdns, mysql and a pretty website [powerdns.com]
djbdns [cr.yp.to] he's grouchy and the no license license thing freaks people out and pisses them off, but people become attached to the quirky but rock solid software.
nstx, ip over dns, yeah... [sourceforge.net]
Re:I R 0wn j00 (Score:4, Funny)
Just askin you big hacker, you.
Re:Never heard of syn cookies or what? (Score:5, Informative)
handshake to set up a connection). DNS uses (primarily) UDP traffic,
which is connectionless (there is no "stateful" connection with UDP).
SYN cookies do no good when your DNS servers are under attack.
run Woody. (Score:2)
Re:A stable version of BIND (Score:2)
Re:A stable version of BIND (Score:2)
Re:A stable version of BIND (Score:2)
I have seen the bind 9 problem - if you are using a single CPU system you may never see a problem but a heavily loaded bind 9 on an SMP redhat system does die fairly often, and leaves a suicide note about a failed assertion. We use a cron job to check bind for signs of life every 3 minutes and restart if need be.
I don't think I've ever seen the problem on SuSE linux though...
Re:MacOS classic? (Score:2)
Re:MacOS classic? (Score:2, Informative)
Re:MacOS classic? (Score:2)
In MacSpeak CDEV is also known as a "control panel" and init is an extension. Those are the little icons that marched across your screen when you were booting up.
By the way, very interesting comments all. Does anyone care to clue me in as to why my original post on this topic was modded completely down? Must admit to being a little clueless on that one. Might b
Re:MacOS classic? (Score:3, Funny)
You mean like a Quadra 950 (~35lbs.) or a pallet of hamburger helper?
Re:MacOS classic? (Score:4, Interesting)
Re:What do they do? (Score:5, Informative)
tm
Re:wtf? (Score:3, Funny)
No, no, no... it's just pronounced "Paul Vixie" but the correct spelling is V-I-N-T C-E-R-F.
Re:Read the fucking article before submitting it (Score:2, Insightful)