Go Daddy: Network Issues, Not Hacks Or DDoS, Caused Downtime 143
miller60 writes "GoDaddy says yesterday's downtime was caused by internal network problems that corrupted data in router tables. 'The service outage was not caused by external influences,' said Scott Wagner, Go Daddy's Interim CEO. 'It was not a 'hack' and it was not a denial of service attack (DDoS). ... At no time was any customer data at risk or were any of our systems compromised.' The outage lasted for at least six hours, and affected web sites and email for customers of the huge domain registrar."
Had to Take the Network Down... (Score:4, Funny)
anonymous U So haxor!!! (Score:4, Funny)
Doesn't Really Help (Score:5, Insightful)
This just makes them look even less competent as a service provider, if the problem was purely internal then.
Re:Doesn't Really Help (Score:5, Funny)
"Good news everyone, we weren't compromised. We're just incompetent!"
Re: (Score:2)
Re: (Score:1)
Actually, the voice in my head is from Icecrown Citadel's Professor Putricide...."Good news, everyone! I've fixed the poison slime pipes!" (during the Rotface encounter)
Re: (Score:2)
Which almost 100% certainly is based on Professor Farnsworth.
Re: (Score:2)
Re: (Score:2)
Re: (Score:1)
"Good news everyone, we weren't compromised. We're just incompetent!"
To be fair, everyone who uses their hosting is already well aware of that, so this doesn't really hurt them to say...
Re:Doesn't Really Help (Score:5, Insightful)
"Good news everyone, we weren't compromised. We're just incompetent!"
And we already knew they were evil [slashdot.org] , so ....
GoDaddy for Congress!
(corporations are people, my friend)
Re:Doesn't Really Help (Score:4, Interesting)
Re: (Score:2)
Or if it turns out it was a hacker, that won't inspire confidence in GoDaddy either.
Re: (Score:3)
Re: (Score:3)
Odd. I saw the reverse: our company's domain which is on GoDaddy's Premium DNS servers was affected, while a colleage's personal domain on the standard DNS servers was unaffected.
Re: (Score:2)
It's not that their premium DNS server is all that robust.
You just can't have secondary slave DNS servers without it...
Re: (Score:3)
Re: (Score:2)
Routing issues aside, the problem could have been largely mitigated with reasonable TTLs IHMO.
I refuse to use GoDaddy because they are lying fucking sociopaths, but from time to time, I do need to help people out with services hosted there. Last time I saw a DNS control panel it had TTLs of 30 minutes as the default for most DNS records.
The outage lasted 6 hours. If you had a TTL of 12-24 hours you would have been just fine, unless I am completely off base on how caching DNS servers operate.
Such short TTL
Re:Doesn't Really Help (Score:4, Insightful)
This just makes them look even less competent as a service provider, if the problem was purely internal then.
It might make them look less competent, but on the flip side suggests an "isolated" incident, and that it won't likely happen again, whereas if it's hackers you as a customer may wonder when the next hack will happen and what effect it will have on your websites, DNS etc. I.e. we better move off before it gets targeted again.
Re:Doesn't Really Help (Score:5, Insightful)
It makes GoDaddy appear incompetent to geeks and computer-savvy users. However, to the average person that does not know much about computers, they will accept it as a computer problem that 'just happens'.... just like all of the errors that they have on their home computers that supposedly have no cause. As long as GoDaddy makes the problem sound really technical while saying they know exactly what caused it and know how to quickly implement a solution that prevents future instances, they will appear competent to the average computer user. After all, to an average user, an admin's ability to solve a problem that sounds complex will make the admin's skill sound really impressive.
Remember many of GoDaddy's customers are individuals and small businesses that have mediocre computer skills that rely on a simplified WYSIWYG tool. To them, evil hackers that steal information are much worse than an annoying problem that just happens because computers all have problems (in their experience). As long as the customer doesn't realize that it was a problem that should not have occurred and it was only caused by incompetence, then they are less likely to lose those customers.
Re: (Score:2)
Re: (Score:2)
then they deserve to have there fucking identities stolen, specially since there using a fucking wysiwyg tool to build a fucking webpage. go anonymous, obliterate godaddy and there fucktarded customers that are to fucking stupid to even exist let alone use a computer.
You tell 'em, Scrappy.
Re: (Score:2)
If you believe that... (Score:3, Insightful)
Then I've got a fully alive not dead elephant to sell you.
Re: (Score:2, Insightful)
How does it help GoDaddy to tell everyone that they are incompetent instead of admitting they were attacked?
Re: (Score:1)
If they were successfully attacked then their both incompetent and insecure.
Re: (Score:2)
And what if that hacker does it again and GoDaddy has to admit they were hacked? They'll lose customer from a second downtime, and they'll likely lose a lot more from being exposed as liars. I don't see a reason for them to lie, really, especially by admitting it was internal.
Re: (Score:2)
http://rs79.vrx.net/works/photoessays/2011/godaddy/ [vrx.net]
Bob makes his own dead pachyderms.
Wow (Score:2)
Talk about having a bad day at the office... can you imagine being a Sys Admin at GoDaddy?
We have a few odd DNS entries still hosted at GoDaddy. We'll be yanking those last ones away. Any advice on how to set it up so I'm not depending on one registrar?
Re: (Score:1)
Any advice on how to set it up so I'm not depending on one registrar?
Yes. You can depend on more hosting services which use different upstream providers and are located in different datacenters. And pray they don't go offline in the same time :)
Re: (Score:2)
That doesn't get around your registrar going down, just your hosting.
Re: (Score:2)
Thanks. I was trying to figure out how cnastase's solution was helping.
Re:Wow (Score:5, Interesting)
You would think so, but the company I work for uses GoDaddy (At least up until today we did, we may be going elsewhere now) for our registrar, but nothing else. We run our own DNS servers, our own web servers and load balancers, our own mail servers, etc. but we got scads of complaints about "the website is down" yesterday during the event. We traced it back to external DNS failures, but I have full-time monitoring on all of our systems and nothing on our end even hiccuped. It worked for some locations but not others.
It makes no sense to me either.
Re: (Score:2, Insightful)
You sure it was not say a SSL issue with certs issues by them - they now are a CA as well. The crl is a crl.godaddy url -- depending on browser, if crl is not available then ssl can be flagged as invalid.
A registrar of a domain has NOTHING to do with the resolving of your domain, once it has been sent to roots.
Do a simple dig +trace query for your domains, where in that line would your registrar be talked too? You hit roots, you hit servers for your tld (org.com,etc), you hit your NS = done. A registrar
Re: (Score:3)
Yes, I'm sure it wasn't an SSL issue. It was a straight DNS "Domain not found" problem.
However, thank you for the idea of looking up the secondary NS records. Turns out our .com's nameservers reside in our .net domain, which is handled by GoDaddy. I'm off to change those to our static IP addresses.
Re: (Score:2)
Bogus glue records pointing to godaddy name servers instead of your own?
Re:Lies (Score:5, Funny)
Never attribute to malice what can be attributed to GoDaddy's incompetence just as easily.
Re: (Score:2)
Never attribute to malice what can be attributed to GoDaddy's incompetence just as easily.
Insecurity and exploitability are forms of incompetence. So this doesn't actually make much of a difference.
Re: (Score:3, Insightful)
What's worse: Not being able to keep your network running when someone actively tries to disrupt it, or not being able to keep your network running under otherwise perfectly normal conditions?
Perhaps... (Score:5, Interesting)
if they'd pay some of that massive advertising budget to competent employees, quality software and proper maintenance. ... naw, bring on the chick ads.
So that's, what, two big hits for Go Daddy this year?
Re: (Score:1)
Go away GoDaddy (Score:1)
I don't buy this for a nickel, and I doubt they are even capable of detecting that a problem was caused by "external influences".
Laughably feeble attempt to save face in front of countless customers who should have known better than to use GoDaddy services.
Re:Go away GoDaddy (Score:4, Insightful)
Hey, cut them some slack. Lying in public is one of the few pleasures of having a customer base that consists of people who don't know better...
Re: (Score:2)
That certainly explains why politicians smile so much...
Re: (Score:1)
I actually do happen to have a clue, Anonymous Coward.
And as I suspected might happen following GoDaddy's cowardly (and patently false) statement, today the "hacker" (who obviously also has no clue?) released some more data he obtained during his attack, to prove that it wasn't a GoDaddy "Glitch". I suspect with their level of incompetence, he probably still has access!
Not so sure (Score:1)
I received emails last night from some SAAS companies claiming that there service outages were due to attacks by hackers. They did not mention GoDaddy specifically, but I know their DNS is hosted there.
Why would they be saying it was hackers?
Sounds like damage control to me.
Re: (Score:2, Informative)
Because someone on twitter said that they did it and everyone believed them?
That makes more sense (Score:5, Insightful)
There was no other indication of a DDOS than the "I did it" tweet by a lone troll. To knock out someone as big as Godaddy for as long as they did would've required an epic-scale DDOS and you'd think something like that would've been noticed by their upstream providers.
This is the second time this week an Anonymous troll lied about an attack (the other one was stealing iPad device ID from FBI)... Anonymous's sterling reputation is being tarnished!
Re: (Score:1)
I like how you so easily believe that they were both false. It makes me giggle.
Re: (Score:2)
We have one Anonymous member claiming that it was his attack that brought down GoDaddy.
We have GoDaddy claiming that it was network issues.
Lacking any further evidence, we're in a he said-she said scenario. While I'm not advocating blindly believing GoDaddy representatives, blindly believing an Anonymous member without any evidence to back him up seems foolish also. (A tweet the Anonymous member made during some service's downtime doesn't count as evidence.)
Re: (Score:1)
What I think you mean is anonymous troll. Anonymous weren't trolling, as in, a person who is an active participant in the Anonymous movement.
There is a difference between an anonymous person and an Anonymous person in this case. The former is just any random person on the internet with a pseudonym trying to troll the latter, which is a group united under one pseudonym.
Re: (Score:2)
What I think you mean is anonymous troll. Anonymous weren't trolling, as in, a person who is an active participant in the Anonymous movement.
There is a difference between an anonymous person and an Anonymous person in this case. The former is just any random person on the internet with a pseudonym trying to troll the latter, which is a group united under one pseudonym.
Maybe you ought to trademark the term "Anonymous" - too many people are using it, customers get confused.
Re: (Score:1)
but then again taking a look at some of the tweets, you get more of a "c'mon are we actually taking this seriously?"
Surely we can take some artistic Licences with the quotes from go-daddy as well! (From the article)
Wagner also noted that the company has provided a 99.999 percent uptime in its D
Re: (Score:1)
Re: (Score:2)
What makes you think an attack would have to involve a DDOS? There are other ways of attacking a network you know.
Re: (Score:1)
TG Posted... (Score:1)
Can't win here. (Score:2)
If you weren't hacked? It shows you have some seriously poor management/backup equipment.
Either way, less people are going to want to use you after this.
Another KKR debacle (Score:2, Informative)
Disclaimer: I work for a company that got "bought out" by KKR, just like GoDaddy.
Since KKR bought them out, GoDaddy has jacked up their prices (to make up for the billions I'm sure KKR "leveraged" out of GD for their execs and shareholders), took a pro-SOPA and PIPA stance (which garnered them a bit of a boycott), and is now having infrastructure problems. I haven't heard, but I'd bet there were layoffs and some brain-drain shortly after the KKR mafia took over.
Go figure.
If you own stock in a company that
Re: (Score:2)
according to www.stockchase.com KKR
"Would characterize dividend as not being safe because it is merely a function of income that the company generates on a quarterly basis. It goes up and down depending on whether they are capitalizing any deals. This quarter will probably have a very nice dividend because they just sold a position in Alliance Boots to Walgreens. Trades at a very reasonable price."
In other news godaddy is still having problems. They weren't relaying emails from a website I have with them, I
Re: (Score:2)
If you own stock in a company that is being eyed by KKR (think: Bain Capital) dump it quick.
Why would you dump stock of a take-over target? Just keep it until the take-over at least... stocks usually rise in anticipation of take-over, because the buyer, no matter how incompetent, will usually pay premium for the stock in order to ensure a quick conclusion of take-over... and if the buyer is so incompetent that he doesn't know that he has to do that, he won't be in a position to ruin the company, because he'll never own enough shares in it.
Well duh ... (Score:4, Funny)
If one "anonymous" person could take down 5 million websites then we might as well give up on the Internets ...
Then again it could have been one GoDaddy Admin who accidentally misconfigured the routing tables that caused all of this ... I'd probably want to be anonymous if I was that person as well ...
Can I have my $80 back? (Score:2)
Good for me, though, since that caused me to switch everything to a competitor and not be affected by this outage.
It's the network!! (Score:2)
Networking is the most often overlooked, often shit on, everyday service that everyone ab(uses) and gets pissed when it doesn't work properly. Like toilets.
99.999% Doesn't mean what you think it means (Score:4, Informative)
They were down for 6 hours but still claim 99.999% uptime. But unless they have been around for more than 57 years, I dont' see how that is possible. Wonder what funky math they use to back up that number.
https://encrypted.google.com/search?q=5+hours+%2F+(1-99.999%25)
Re: (Score:2)
I suspect they declared a 'maintenance window' 5 minutes into the outage.
Re: (Score:1)
I read somewhere their 'premium dns' service was unaffected - not got a link though.
Re: (Score:2)
I don't buy that. It may have appeared up, but that's because everything was intermittent all day.
So what? (Score:3)
To this I say "so what"? When you have one primary job to do - respond to DNS requests for millions of domains that are registered through you - and you fail to do so, it's over. No matter what the root cause is, you caused *millions* of web sites to be unreachable for most people, for a period of time spanning hours. This is not "oops", this is catastrophic failure from a business perspective.
I can only hope that sufficient numbers of customers will be as offended, and seek more reliable solutions.
Intermittent (Score:3)
"Yesterday, GoDaddy.com and many of our customers experienced intermittent service outages starting shortly after 10 a.m. PDT. Service was fully restored by 4 p.m. PDT. "
http://www.godaddy.com/newscenter/release-view.aspx?news_item_id=410 [godaddy.com]
Must be that new definition of the word "intermittent." The one roughly synonymous with "total."
but what if the cause *was* a hack? (Score:2)
A lie, but... (Score:2)
anonymous? (Score:1)