EveryDNS Under Botnet DDoS Attack 154
mellow marsh writes "EveryDNS, sister company to OpenDNS (which runs the PhishTank anti-phishing initiative), has been hit by a massive distributed denial-of-service attack. The attack started sometime Friday afternoon and, from all indications, was targeting Web sites that used free DNS management services provided by EveryDNS. At the height of the DDoS bombardment, EveryDNS was being hit with more than 400mbps of traffic at each of its four locations around the world. From the article: '"We were collateral damage," Ulevitch explained... Because law enforcement is involved, Ulevitch was hesitant to release details of the actual target but there are signs that some of the targets were "nefarious domains" that have since been terminated.'" OpenDNS, which makes use of EveryDNS services, was affected for a time, until they spread their authoritative DNS more broadly. The EveryDNS site is now reporting that the attack is continuing but has been mitigated and is not affecting operations.
puppy (Score:5, Funny)
Re: (Score:2)
COM != NET (Score:3, Informative)
Re: (Score:1)
Re:COM != NET (Score:4, Informative)
Another quality, editor approved Slashdot story. Great job, guys.
Affected; Irony (Score:2, Interesting)
My comp sci networking class assignment was on my home server, and I use EasyDNS. Had to bus home and put it on a USB stick. Last day of class, and the end of a particularly brutal week.
Re: (Score:2, Funny)
Re: (Score:2)
Save some time and punch in the IP address instead of a URL and skip the DNS lookup.
It's your server. Do you know your IP address?
Re: (Score:2)
Yes, however many ISP's do not reassign IP's very often except those using PPPOE which may "dial up quite often"
correct URL (Score:4, Informative)
Re: (Score:2, Insightful)
Does that mean (Score:2)
Heh (Score:5, Informative)
I'll keep it up for Slashdot, let me just move it around a bit.
-david
Re:Heh (Score:5, Funny)
Re: (Score:2)
Link To Them (Score:2, Funny)
This is nothing short of organized crime (Score:1, Troll)
Re: (Score:2)
Questions? (Score:5, Informative)
If you have questions about this or DDoS in general, feel free to ask them here and I'll make sure to cover them in my response. I'll be writing about what we've seen and what I generally do when it comes to soaking up traffic and how we handled this event in particular. (The short answer: find the smartest people you can to help you and then start taking corrective action)
Thanks!
David Ulevitch
Re:Questions? (Score:4, Insightful)
Was this a 'righteous' attack on malicious websites?
Or just some intramural warfare by one nefarious group upon another?
Re:Questions? (Score:5, Interesting)
You must be new here (Score:2)
A Slashdotting is always Righteous
/Don't hate me because my UID is prime
Re: (Score:2, Funny)
Re: (Score:1, Interesting)
2) This might be harder to tell, but what type of clients were hitting you (high speed home users, commercial end servers)?
3) The poster said " 'We were collateral damage,' Ulevitch explained..." How so, and who was the primary target?
Its not all too bad, just 4 days ago, I found out about OpenDNS. Great stuff, gave me a solution to my horrible ISP's (Charter Comm.) DNS servers. And until I saw this post, I didn't know about EveryDNS. Hopefully this
Re: (Score:2)
Re: (Score:2)
Global Stats:
Accounts: 62357
Domains: 103552
Records: 292615
The implementation details are in the FAQ and About. Without bothering to read them again, I think they use a modified tinydns.
Re: (Score:2)
Re: (Score:2)
Re:Questions? (Score:4, Informative)
That's less trivial to filter, especially when your upstream isn't being cooperative. In our case, which you'll read about tomorrow or Monday, we quickly were able to jump onto a network run by some folks with very very high levels of clue; nLayer operated by Richard Steenbergen. Their website is cheesy -- don't let it fool you. They are a seriously run network providing transit across the country to a bunch of other networks. Check routeviews for proof.
-david
Re: (Score:2)
As a sidenote, everydns hosts macports.org which was affected by the DDOS. Even though macports also had two other working DNS servers besides the 4 from everydns, I had
Re:Questions? (Score:5, Interesting)
1. How did you manage the response? The one-smart-person-in-charge-who-stays-awake-the-wh
2. What tactics worked, and even more important, what didn't work?
3. What sort of agreements should people have in place with their upstream ISP prior to an incident?
4. How intelligent was the attack traffic? Randomized payload? Does anyone bother spoofing addresses any more?
5. Was it a guided attack or a fire and forget? In other words, did the scum make any changes to their tactics in real time as you tried corrective action?
6. What if anything can be done in the first few minutes/hours?
7. If you had to choose between capacity and filtering, which would you choose?
Re: (Score:2)
Real ripple effects, even from this small event. (Score:5, Insightful)
Worse, the state government box's spam filtering appliance blacklisted the retailer's server, and a third party admin had to get involved to free things up. Quite a mess.
But the real lesson? People who say that a "cyber attack" couldn't really hurt the economy are wrong, wrong, wrong. This stuff can be really disruptive, and this was a pissant little scaled-down example. No major damage, but a lot of thrashing around, untold manhours of lost productivity, and (in the case of the anecdote in question, involving just one retail company), probably some tax fines which will require much tail chasing to get waived once the the story is clearly told, assuming the state government in question is feeling sporting about it.
Re:Real ripple effects, even from this small event (Score:2)
I could cause a lot more problems and not do anything illegal. Shoud those acts be illegal because of a butterfly effect caused by bad programming? Get real, please.
Re: (Score:2)
If by "bad programming" you mean: the DDoS attack on the name servers was working, and thus a receiving mail server couldn't decide whether to trust another party's sent message... then, sure. Except that's not bad programming "on the site" (as you put it), is it? No. It's a vulnerability in using DNS in the first place. The only thing that would have prevented that would have been sticking with good old
Re: (Score:2)
Re: (Score:2)
I'm a little mystified at how you come up with this, but just to be clear: all I'm pointing out is that, as we sit, the proverbial "cyber attack" CAN indeed cause considerable economic disruption. I'm not sure what you think I'm hiding behind when I say that. It's just a statement of fact, and this one little event shows how disruptive it could be. I've made no particular call to action, but I certainly wouldn't mind if people who use bot-nets to cause
Re: (Score:3, Insightful)
Learn to spell, get a clue.
There is nothing you can practically do to prevent someone on the internet from sending a packet addressed to you, nor two packets, nor 1000000. There is nothing you can practically do to prevent the source address on each of those packets to be different. If a DOSer has much bigger pipes than you, you are sunk, unless you can do something very smart. For a start, getting remote access to your server during a DOS attack is tricky unless you have re
Re: (Score:2)
And at that point, it becomes a question of how you deal with a DDO
Re: (Score:2)
Right. And with GOD on your side, armageddon becomes something interesting to watch while having some coffee.
Pop quiz,
Re: (Score:2)
Always being offline costs nothing (unless one is stupid enough to pay for it).
Those two cases are easy, boring and unrealistic. The interesting stuff is in-between.
Re: (Score:2)
Prevent? How? "Don't hang out on IRC from your server's IP?" "When you get an email demanding $50000, pay up?" "Reach through the intertubes and strangle the guy that's about to send the packets to you?"
They might be "easy to deal with": call your upstream provider and hope that they'll shut it off (or call their upstream provider) rather than go "kaching!" and let your bandwidth bill rack up.
Re: (Score:2)
incompetence effects, not ripple effects (Score:4, Insightful)
In once case, a vital piece of mail sent to a state taxing authority couldn't get through on a month-end calendar deadline, causing much grief.
Maybe a)it shouldn't be left until the deadline and b)sent via email, if it's so damn important.
And maybe you not tell clients to use a free DNS hosting service as their sole DNS provider...
Re: (Score:2)
Hey! I don't do management consulting for their accounting people. But sometimes this sort of thing tends to have that effect, once the dust settles.
And maybe you not tell clients to use a free DNS hosting service as their sole DNS provider...
Not my call on this one either. Our team is involved on a peripheral project, and this part of their infrastructure was in place long before we got on board. We've
Re: (Score:2)
In most/all companies, month-end and year-end are major periods of effort for accounting to close the books. Some of that includes communicating information with third-parties...
Re:Real ripple effects, even from this small event (Score:2)
Re: (Score:2)
I believe the official policy is that things are supposed to take place by postal mail, and FAX by fallback. But folks at both ends had been swapping mail for months with no problem (and more reliably AFTER the spam filtering went in), and got seduced into assuming it would always work. That's what happens, I see
They deserve the grief (Score:4, Insightful)
Myself, a month ago I missed an opportunity to collaborate on a TV miniseries. Why? Because the moron who asked me for my collaboration absolutely trusted e-mail, and it was **THE** message that bounced thanks to a network glitch, and that moron didn't think of calling me on the **PHONE**. Well, if they were stupid enough to trust e-mail like that, they probably would have made a crappy miniseries anyways.
For casual communications, there is e-mail.
For vital ones, there is registered mail, fax or phone.
Re: (Score:2)
Yup. But when (in the case I'm citing) an accounting type and a person at a tax office have been happily swapping mail for many months, with little or no lag, they tend to get lulled into a sense of false reliability. And that's what happens.
Re: (Score:3, Insightful)
A lot of mail is misdelivered or just lost. Yet the tax people do not demand that we send in our tax returns by registered mail. And would you be as pissed at the miniseries people if they'd sent you a letter by regular m
Re: (Score:2)
I'm sorry, this is slashdot. You are not allowed to use reasonable, constructive analogies to make a point. Also, you should sound just a little more hysterical, and be sure to somehow blame Steve Balmer if at all possible.
Thanks, though! I sho
Re: (Score:2)
Are there any e-mail style systems that do claim to be 100% fail-proof?
But not for that reason (Score:2)
Relying on a single third party DNS service is pretty stupid if reliability important.
Redundant links, geographically and geopolitically dispersed DNS, careful administration.
Engineer reliability between all important endpoints.
Do that and you can send rather important things via e-mail and be confident that they will get there on time.
Be sloppy and it's no better than relying on a cell phone with poor coverage, a weak battery, and a pre-paid plan.
"nefarious domain" is a loaded and subjective term (Score:5, Insightful)
to some.. the pirate bay and allofmp3 are "nefarious domains"..
to others "www.f**Ktimewarner.com" and "walmartsucks.com" are "nefarious domains"
and to others "www.wikipedia.org" and "www.aclu.org" are "nefarious domains".
I have a lot of trouble with the idea that DDOS attacks were being carried out in (apparently successful) attempts to wipe domains off the face of the earth..
this implies the attackers had no legal standing to take those domains offline.. then they call them "nefarious" after the fact.
Re:"nefarious domain" is a loaded and subjective t (Score:1)
What exactly being wicked would depend on the situation (as that's a subjective term) and considering that they are trying to take down websites via DDOS attacks, I'd call that wicked.
Although, I don't understand your last statement. Is it wrong to call them nefarious after the fact? Wouldn't you call a person a murderer after they murder someone?
What's the motive? (Score:1)
Re: (Score:3, Insightful)
It's an indirect attack against people who use EveryDNS to get traffic to their own sites (or mail servers, etc). If you ran, say, an online casino, and your main competition for a particular type of customer happened to have EveryDNS doing their forward lookups... and you could shut down your competition for at least a full business day by torpedoing the DNS they need to be seen - presto, done.
Botnet? Cal it what it is! (Score:4, Insightful)
Where are the class action suits against Microsoft for continually producing such flawed software that makes it easy to 0wn a box?
If it wasn't for 20 some years of MS indifference towards security, there wouldn't be botnets like this, being used for DDOS attacks and forwarding billions of spams a day.
Re: (Score:1)
Re: (Score:1)
Uh, wrong. No software is completely secure, especially something as complicated as an operating system. This would still be happening, except it would be on Unix/OS2/Apple boxes instead of Windows. Get over it.
Re: (Score:2)
This of course doesn't help the remote exploits, buffer overflos [in file formats] and other problems that are totally native to MS [and go unfix for random amounts of time]. Not that bugs don't happen in the OSS world, but they tend to be fixed faster, and a larger portion of OSS users are more aware of secure computing practices [e.g. not running as root, not opening every f'i
Re: (Score:1, Insightful)
Me, I'm a geek who uses Windows. I do computer fixes for a living, and I need to be intimately familiar with the systems my clients use, which are almost 100% Microsoft. Yet, despite running such 'inherently flawed software,' I haven't had a virus/trojan on one of my boxes. Ever. Turns out geeks are secure, regardless of what OS t
Re: (Score:2)
What's that? FreePartyPoker.net? All I have to do is download some random
So MSFT obliges them, running as root to avoid those pesky "user privilege escalation" prompts, ActiveX scripts and downloads (which the stupid users with no tra
Re: (Score:2)
Society has somehow forgotten that people were using computers just fine 25 years ago without the need for shiny GUI running as root.
At least in Apples respect they have merged their nice GUI and applications with a well tuned Kernel and userland. My only gripe (other than the cost) is the crappy 4.0.1 GCC that comes standard with
Re: (Score:2)
Tom
Re: (Score:2)
Sure, I wouldn't expect all computer users to be C experts, but they should understand the basics of a file system (e,g. files, permissions, directories, symlinks, etc) so they can properly interpret what they are looking at and doing. Not running as root is another good start, KNOWING WHY you shouldn't is better
Once you beat the "oh I can't learn this" stubborness from most people you'll find that they can be taught the basics fairly easily. It'
Re: (Score:2)
Re: (Score:2)
What is this, "The Preview Button", you speak of?
These words confuse me!
Re: (Score:3, Insightful)
Re: (Score:2)
thanks for the correction.
Re: (Score:2)
What do you think antivirus and antispyware apps do? On Linux I had to track it down by hand.
Which will be 0wned after 30 minutes connected to the Net, due to the POS that is Windows, coupled with the ignorance of the bog-standard Windows luser.
Windows XP Service Pack 2 won't be 0wned just by connecting it. And if you're gonna throw user ignorance into the m
Re: (Score:2)
Re: (Score:3, Insightful)
Re: (Score:2)
Can you say with certainty that whatever OS people would be using instead of Microsoft would have prevented this?
Re: (Score:2)
Open Letter to all Trolls (Score:5, Interesting)
Nothing positive or lasting will come out of trolling (and yes: this means you anonymous asshats on
So why not be part of a winning team and stop script kiddie'ing around from your parents basement.
Sincerely,
The Rest of the Human Race.
Stupid Me... (Score:2)
Re: (Score:1)
Possible Target? (Score:1)
Re: (Score:2)
Every DNS, not EasyDNS. (Score:2)
I have to stress that it is EveryDNS that is under attack, and not EasyDNS.com [easydns.com].
That being said this is not an uncommon issue these days at DNS providers across the 'net. Before anyone starts to kick and scream about how EveryDNS is handling things, remember that these attacks can get astoundingly vicious.
No amount of "clue" or mitigation or whatnot will help when the upstream service providers themselves are having trouble with the traffic load from a large-scale botnet attack.
Re:Every DNS, not EasyDNS. (Score:4, Informative)
Not to mention that networking people generally don't give a shit about bandwidth- it's packets per second that kill routers, not bandwidth. Assuming 100 byte packets that's about 4Mpps- Even a basic 7600 can handle this kind of traffic. Assuming 30 byte packets (can't be smaller than that) you're talking about 15Mpps. Again Even a basic 7600 should be able to handle that- not to mention a Juniper M7i or similar. Most Foundry equipment would laugh at that rate. All of these routers can do ACL's at full packet rates.
That said- other recent DNS attacks exceeded 1.5 Gigabits per second of traffic and were a lot more vicious than the attack being described here.
I'm not knocking EveryDNS- I know what a bitch dealing with a DDoS can be- the problem tends to be that most people aren't ready to deal with it. Using BGP community based nullrouting most service can be restored within seconds of the target IP(s) being identified. That allows admins to keep untargeted systems and services up while the attacked systems are dealt with. The admins can then use the time to locate some/any pattern in the attack or enable the appropriate filtering such as a Cisco Riverguard or similar.
-sirket
Re: (Score:2)
Attacks like this seem not to be targeted at taking the provider down forever technically, but making it so expensive for the provider to continue to operate
sue (Score:2)
if you have a dog and it bites someone or damages someone's property you are liable, so why not computers?
Re: (Score:2, Insightful)
Re: (Score:2)
The owner could be sued even if the dog has rabies.
Re: (Score:2)
A. You allowed your dog to be in the open enough for someone to infect it.
B. You refused to notice different behavior patterns in your dog (ie, he was slow to respond, seemed to be preoccupied every time you called him).
C. You refused to take your dog to the vet often enough to notice this distemper, and he bit someone.
You are liable. Replace dog with PC, and vet with "antivirus" and we're all good.
DNSPark, too (Score:3, Interesting)
At least...! (Score:2)
Thank you David! (Score:2)
And a reminder, EveryDNS.net runs on donations.
EveryDNS Donations [paypal.com]
Thank you again.
ps: Wow, slashdot uid 18.
I wish everydns had SPF support (Score:2)
Re: (Score:2)
I would hope so. That would be 400/1000 bits of traffic per second. ITYM Megabits.
Re: (Score:2, Informative)
Re: (Score:3, Funny)
Re: (Score:1, Insightful)
The difference is that very few people knew the exploits and fewer still were in a position to actually use them.
Re: (Score:2)
Secondly, the Morris worm compromised _Unix_ systems- not windows boxes. I loathe Windows but please don't pretend Unix doesn't have it's own sins. The difference is the Unix folks tend to learn from their mistakes and the Windows folks don't.
-sirket
Re:solution to DDOS attack (Score:5, Informative)
First off- be prepared for a damned attack and don't wait til it happens. When an attack does come:
1- Identify the target IP address
2- Immediately null-route traffic for that address (preferably using BGP community based null-routing)
This gets the rest of your systems back up and gives you time to work on the problem.
3- Try to identify a pattern in the attacking traffic- use a product from a company like Mazu- or just tcpdump if you're good with sed and awk.
4- If there is a pattern ask the upstream ISP to block based on that pattern (same source port, same source IP, same TTL, whatever). Or block it yourself if you have the router and bandwidth capacity to deal with the attack yourself- though that's generally a waste of your resources.
5- If there is no pattern but the traffic is malformed then enabled a Cisco Riverguard or similar protection device that can filter out malformed traffic at the higher protocol layers. As an alternative, sign up for such a service form a company like Prolexic.
6- Remove your null route and see how you did.
7- If you can't afford a protection service, you can try moving the host/dns records to new IP's. Sometimes the attacks don't follow- sometimes they do. It's often worth a try as it can be done faster than enabling protection services in many cases. In this case leave the old null route in place until the attack stops. Be prepared for the attack to return at any time once they realize what's happened.
Make sure to keep traffic logs for law-enforcement and to share with other ISP's so that they can track down the offending bots.
In the future try to keep your traffic as segregated as possible such that an attack on a single host will not take down too many other services should you need to null-route that address for an extended period of time.
The easiest solution- block all IP addresses assigned to the APNIC region and watch as your site immediately returns to normal. Sadly most of the DDoS's I've seen recently had the majority of their traffic sourced from APNIC addresses.
-sirket
Re: (Score:2, Informative)
It's a *distributed* attack. That means more than one address. A lot more.
'or just tcpdump if you're good with sed and awk.'
You're going to be able to do this on 1.6Gbit of traffic in realtime? That's good typing.
'The easiest solution- block all IP addresses assigned to the APNIC region and watch as your site immediately returns to normal.'
FUD. This is a botnet attack. Most owned PC's live in the US. It's this kind of thinking that has forced us to run our servers in the U
Re: (Score:2)