Dark Day In the AWS Cloud: Big Name Sites Go Down 182
An outage of one company's servers might only affect that company's customers — but when a major data center for Amazon hits kinks, sites that rely on the AWS cloud services all suffer from the downtime. That's what happened today, when several major sites or online services (like Instagram and AirBnB) were knocked temporarily offline, evidently because of problems at an Amazon data center in Northern Virginia. From TechCrunch's coverage of the outage: "The deluge of tweets that accompanied the services’ initial hiccups first started at around 4 p.m. Eastern time, and only increased in intensity as users found they couldn’t share pictures of their food or their meticulously crafted video snippets. Some further poking around on Twitter and beyond revealed that some other services known to rely on AWS — Netflix, IFTTT, Heroku and Airbnb to name a few — have been experiencing similar issues today."
Running List of Cloud Outages? (Score:4, Insightful)
I thought this might already exist, but I'm not finding it with a quick Google search. Seems like it's a thing that could get ad views from some decent IT audiences.
Re:Running List of Cloud Outages? (Score:4, Funny)
Self Documenting? (Score:2)
Is that recursion or just self documenting?
Re: (Score:3)
http://sitedown.co/ [sitedown.co]
for Amazon, it is http://sitedown.co/amazoncom [sitedown.co]
watch out for birth rates (Score:2, Funny)
When morons can't watch TV (or equivalent) they fuck. 9 months later you'll see a birth rate spike.
Add Adobe Creative Cloud to the List too (Score:4, Interesting)
That went down and I think it ate some files with it. Just before the crash my client reported 103 files being removed. They weren't by me.
Where are the NSA comments?? (Score:3, Funny)
Re: (Score:2)
What keeps big data clinging to the Eastern USA?
Re: (Score:2)
population density, closeness to physical infrastructure, larger pool of qualified workers (maybe), etc.
Re: (Score:2)
According to our copy of your files you got it all right, need us to forward it again?
Yours,
NSA
Has Rackspace had any outages in 10 years or so? (Score:5, Interesting)
I've run servers on both Amazon and Rackspace for several years now and I can't recall a single instance of Rackspace having an outage. On the other hand, Amazon seems to have major issues at least 2 or 3 times a year. Is this stuff tracked anywhere?
Re: (Score:2)
Re: (Score:2)
Netcraft?
Re:Has Rackspace had any outages in 10 years or so (Score:5, Informative)
Re: (Score:2)
I was a former Slicehost user in the St. Louis data center and then was moved to Chicago after the Rackspace acquisition. Even so, there's never been so much as a blip from there in the last 5 years. Probably is data center dependent, I just never remember hearing about anything.
Friend of mine here in town owns a web business using about 9 Rackspace servers to host 700 websites and he said they hadn't had an outage in the last 8 years.
Amazon Storefront Problems? (Score:2)
Multiple regions, anyone? (Score:2)
Isn't this why AWS offers multiple regions?
Such large sites should understand that having multiple availability zones means nothing if the zones are all in the same region. Oh, and your application would need to be designed for failover.
In addition, when looking for high-availability, you don't segregate your audience to individual regions. You let the working regions take over for you.
Or spend the extra money and set up your own co-lo arrangement.
Everybody that is surprised is stupid... (Score:5, Insightful)
That things like this will happen with a cloud infrastructure are obvious. That the reliability claims made by the cloud providers are fantasy is also obvious. As soon as they start to do "uptime or else" (meaning you get tons of money as downtime compensation), things may be different. but they will not do that. At this time, the only thing you can do is change to a different cloud provider, which will have the same issues. Uptime guarantees without penalties when failed to meet them are worthless.
Re:Everybody that is surprised is stupid... (Score:5, Insightful)
We built a decentralized network called The Internet, even capable of withstanding global thermonuclear war -- packets rerouted moments after a city disappears from the mesh... And folks use data silos? Protip: Don't centralize services, that's daft in terms of both uptime and congestion.
Re: (Score:2)
The internet ain't what it used to be. The internet of today couldn't withstand a punk with a BB-gun, let alone a tacnuke.
Re: (Score:2)
Our contract at data center that we host at has significant penalties for downtime [qualitytech.com]. In about 6 years of hosting there, we've had exactly 2 incidents of less than 1 hour each.
Of course, the deluge of notifications we get every time a fly causes a ballast to fail in the 3rd light down the main hallway, or when our network usage at 95% exceeds the monthly average by 0.05% get a bit annoying, but I have no complaints of the quality of service.
Re: (Score:2)
Seems some people are getting it right. No surprise. One reason the "cloud" is cheap (well, it is not really if you look closely enough), is that it cuts corners that cannot be cut when reliable operation is needed.
Re: (Score:2)
They say "99.95%" in their SLA (http://aws.amazon.com/ec2-sla/), but only as target and only when "commercially reasonably". That is commercially reasonable for them, not the customer, and that is the real problem. If you run on your own infrastructure, commercially reasonable refers to what it costs _you_ when _your_ site is unavailable. That is a whole different thing. You are right, that there is some compensation, but it is topped at 30% cost reduction and hence completely laughable.
This is why I laugh at tech pundits who preach... (Score:4, Interesting)
Re:This is why I laugh at tech pundits who preach. (Score:4, Interesting)
Depends on which "future" you are talking about. The future where the bulk of personal data is stored on the cloud to be shared across devices and with friends, family and authorized services is one I think is bound to come to fruition.
The future where Corporations put their core infrastructure into the Cloud is not one I ever recall anyone talking about.
Wrong terminology? (Score:5, Funny)
Serves you right (Score:2)
For believing and investing in some handwavy concept called 'cloud' where you abrogate responsibility take the iOS view (it Just Works) of technology.
Pretend this was a US government outage (Score:4, Insightful)
The right wing talking heads on TV would be squealing like stuck pigs. They would be screaming about "gubment" waste and incompetence, and start floating bills to privatize the FAA (or whomever). You'd get the same response on Slashdot as well.
Meanwhile in real life AWS, Google, and NASDAQ have all had dramatic failures in recent weeks. Although NASDAQ got a fair amount of coverage, and Google got some mention, AWS has been pretty much below the radar for the mainstream media. No one is making dramatic statements on TV about how Google is run by a bunch of idiots, or NASDAQ, a quasi-governmental entity, should be nationalized, because when it fails the entire economy is as risk. As far a critical comments, it's the sound of crickets.
Clearly, there is a double standard. When there are problems with technology in the public sector, it's all hostility and table thumping. Similar failures in the private sector are treated like natural disasters completely beyond human control. According to common rhetoric, the private sector is always better then the public sector. Yet when the private sector fails, no one ever compares it to the well functioning public sector.
There is clearly a lot of hypocrisy in bashing the government. A lot of political power is at stake, and along with that goes a lot of money. This situation makes some people very happy, because they are getting what they want, both in public policy and private profit.
Re: (Score:2)
pretend it was the FAA having a big chunk of airspace loose all ability to track aircraft, or NOAA loosing data collection so that weather forecasts are disrupted...The right wing talking heads on TV would be squealing like stuck pigs. They would be screaming about "gubment" waste and incompetence.
Because its their money being wasted.
Meanwhile in real life AWS, Google, and NASDAQ have all had dramatic failures in recent weeks. Although NASDAQ got a fair amount of coverage, and Google got some mention, AWS has been pretty much below the radar for the mainstream media. No one is making dramatic statements on TV about how Google is run by a bunch of idiots, or NASDAQ, a quasi-governmental entity, should be nationalized, because when it fails the entire economy is as risk.
The people who care (i.e. people who were hosting at US-East-1) know, and they have the opportunity to withdraw their custom from AWS. They can employ another provider, or bring it in-house and do it themselves.
Clearly, there is a double standard.
No, you are just comparing apples and oranges - people don't bitch about private companies because, worst comes to worst, they can take their custom elsewhere. Government needs to be held to a higher standard precisely because that freedom is lacking.
Re: (Score:2)
Because its their money being wasted.
To some extent. But it is frequently pennies of their money that is being wasted, if that much. They make it sound like they're the sole supporters of whatever government agency had an issue.
No, you are just comparing apples and oranges - people don't bitch about private companies because, worst comes to worst, they can take their custom elsewhere. Government needs to be held to a higher standard precisely because that freedom is lacking.
In quite a number of instances, you can't - or at least, there is no comparable product to select. Example: Internet access, health insurance, airport, airline, power, etc.
And government isn't being held to a higher standard, government is assumed to be incompetent by definition. Not worse, not less efficient, but liter
nothing of importance affected (Score:2)
entertainment and social media down? who gives a shit? grow up.
Best week ever for sys admins (Score:2)
I have to say with all of the big names having problems recently this has been one of the best weeks ever for the lowly corporate sys admin. Now if the company's email, file or web server--or even the coffee machine goes down, they can point to the big names that also have problems. It's great to be able to say that even at companies like Amazon, Google or Microsoft with all of their talents their servers also have problems. It's the greatest excuse ever for tripping over the power cord. And if that doesn
Re:Say what you will (Score:5, Funny)
In Soviet Russia, company's customers go down on YOU!
Re:Say what you will (Score:4, Funny)
In Soviet Russia, company's customers go down on YOU!
so dirty...
Re:Say what you will (Score:5, Funny)
"In Soviet Russia, company's customers go down on YOU!"
Now we know the truth about why Snowden went there...
Re:Say what you will (Score:5, Interesting)
One of the features of AWS was supposed to be the ability to reroute everything to a different datacenter if one goes down. I know I read that somewhere back when AWS was first starting up. You don't think they lied, do you?
Re:Say what you will (Score:5, Insightful)
Re: (Score:2, Interesting)
Why the lack of power and real optical links that where regional, power distinct.
Is this like the idea of linking to a site/city/state/regional 'ring' many times? Very safe from any local cut/drop, cheap, but still very dependant on one geographic provider?
You also have a submarine communications cable (France to the USA) on the way for that State??
Re:Say what you will (Score:5, Informative)
either you don't speak English or you need to take your meds. no offense. so i'll try muddling a reply together for you.
There are many ways to setup remote failover systems. Most of them rely on some type of heartbeat system where there's a "heartbeat message" which they all send each other periodically, and if the current Active goes out of response for too long the others choose one to take over. So it doesn't matter if they're all in one room connected with a single switch, or spread all over the planet.
The real rub for any mechanism is DNS... if the primary server your FQDN points at drops then you might have redundancy but most people won't be able to take advantage of it. With more manual mechanisms (such as telling users "If our primary site goes down, try here instead!") that's not as much of a concern, just a PITA to keep track of.
Re:Say what you will (Score:4)
yeah, but cloud is sold as this super cheap way to compute and have five nines reliability
actually, no (Score:5, Informative)
"cloud" is sold as a *convenient* way to compute, where it's quick to add resources when needed so you can start small and scale up (and down) with demand.
It is *not* generally considered a cheap or particularly reliable solution. So far at least none of the cloud providers are offering five nines--if you want that, you should (for now at least) jbe looking at enterprise/telecom gear.
Realistically (Score:3, Insightful)
Chances are that there are no providers that offer a true 99.999% uptime. If you demand that, you need to be building your code to run in a HA cluster with nationwide dispersion. (For reference, you get 5.25 minutes of downtime across a whole year).
99.999% uptime is also completely unnecessary, but sounds really good to management until you talk cost.
Re: (Score:3)
Chances are that there are no providers that offer a true 99.999% uptime. If you demand that, you need to be building your code to run in a HA cluster with nationwide dispersion. (For reference, you get 5.25 minutes of downtime across a whole year).
99.999% uptime is also completely unnecessary, but sounds really good to management until you talk cost.
Just make sure that scheduled downtime isn't included in the 99.999% and schedule downtime on a regular basis!
Re: (Score:2)
How up time is calculated is one of the really weaselly ways that companies set up SLAs. Some companies don't start counting downtime until it's reported, others require a minimum threshold of downtime before it counts, others define available in somewhat meaningless terms (e.g., server up, but network down doesn't count).
I'd like my 911 service to be reliable... (Score:2)
There are some things where five-nines makes sense.
Disclaimer...I have worked in the telecom industry in the past.
Re:actually, no (Score:4, Informative)
Quoth Forbes:
Cost savings… [...] These are the advertised benefits of cloud computing
Quoth Salesforce:
4. Cap-Ex Free [...] no need for capital expenditure [...] minimal project start-up costs
Quoth Verio:
Achieve economies of scale [...] Reduce spending on technology infrastructure. [...] Globalize your workforce on the cheap [...] Reduce capital costs.
And those were the first 3 hits for ``benefits of cloud computing'' (although the first one is meta, it refers to others refering to cost savings).
I hate to shake you from your firmly entrenched world-view, but you have to know that people are touting cloud solutions as ones which have cost benefits. Whether they're valid claims or not is irrelevant, they are undeniably being made.
Re: (Score:3, Funny)
YOU sir look like a shrewd and discerning businessman. How would you like to buy a bridge?
Re:actually, no (Score:4, Insightful)
Whether they're valid claims or not is irrelevant, they are undeniably being made.
Like "core business" Every time I hear that, it's from a contractor or someone who just spoke to a contractor, and it's always about why it's good to outsource everything to contractors. It doesn't take long for that to be a pattern.
Now, ask cloud computing companies how much they charge, compared to renting tin. It's always cheaper, except when it's not, and even then, it's cheaper to use the more expensive cloud because tin can go down, the cloud can't, or something like that.
read carefully (Score:3)
"no need for capital expenditure" and "minimal start-up costs" are not the same as "cheap". All it means is that you don't need to pay up-front.
It's like renting a car for a day vs buying one. If you only need a car a few times a year, renting is cheap. If you need a car every day for a decade, you should probably buy one.
Re: (Score:3)
...nothing it does is new...
Ahem, it siphons additional funds from customers ;-)
Re: (Score:3)
When you want multiple locations("regions" in aws) you need to pay for resources in each additional region, then pay another cost to provide that failover.
Or you can have storage in those regions prepped to failover, with no other resources provisioned. When failure needs to occur, you start spinning up the instances in the other region.
It does require planning; you can reroute But you don't get that automatically; it requires work and preparation.
Re: (Score:2)
Configure the warm resources in the other region to constantly monitor the primary. If the primary goes down, they automatically activate the secondary.
Re:Say what you will (Score:5, Insightful)
"nothing it does is new or cheaper than hosting 10 years ago."
Welcome to the wonderful world of marketing. Sell people what they already have for 50% more.
Re:Say what you will (Score:5, Insightful)
No they didn't lie. You can set things up that way-simply set up your servers in multiple data centers(AWS availability zones) and load balance between them. It's foolish to just throw things up in the cloud and think magically I won't ever have to worry about downtime ever again. It's foolish-but a lot of companies act this way.
Somehow cloud hosting is taken as the silver bullet to prevent outages-it isn't. You still have to architect things the way you would normally if you're looking for things like disaster recovery, high availability, etc...etc..
Re: (Score:3, Insightful)
No they didn't lie. You can set things up that way-simply set up your servers in multiple data centers(AWS availability zones) and load balance between them. It's foolish to just throw things up in the cloud and think magically I won't ever have to worry about downtime ever again. It's foolish-but a lot of companies act this way.
But that's the problem. *THEY* (i.e., AWS or whoever) are supposed to take care of all that stuff. They're supposed to worry about "uptime" and fixing things when they break and having redundant systems that kick in when something breaks so that there's no loss of service. That's the whole point of putting stuff in the "cloud".
If * I * have to worry about that stuff then I might as well just do it myself and not give my money to Amazon.
Re: (Score:2)
However, AWS provides tools for developers to do it.
Re: (Score:3)
Oh it's possible to do just rather expensive to do well. Disk based bits don't work as sync writes past a region take far to long. Higher up the stack you can deal with 35-70ms of network latency. But now it's not mysql with any old crappy php code.
AWS is PHB buzzword like IBM a decade and a half ago it makes the VC guys happy that you fixed your scaling issue. In reality everybody else's scaling issues now impact you.
Re:Say what you will (Score:5, Interesting)
AWS is really great at scaling. It's better than anything else on the market, but it does require a lot of work.
Re:Say what you will (Score:5, Interesting)
No, you have to manage your own redundancy and failover on AWS. Look at all the effort Netflix has put into programming failover and stress testing and yet they still have frequent outages with AWS.
Re:Say what you will (Score:4, Informative)
AWS Status Dashboard?
I know this is /., and people here don't like to read, but did anyone actually read the status dashboard posts?
This issue was limited to a single AZ, effected only a small number of machines, and was specifically an issue with added latency in EBS volumes. And Amazon completely resolved the issue in 4 hours.
So, call me crazy, but didn't they do exactly what they are supposed to do? Also, AWS quite clearly states that any given AZ *might* fail. Hence, if you want any sort of high-availability, you replicate across different AZs.
Plus, I have 10+ EC2 instances, and a number of other resources with AWS, and none of them were effected by this outage.
Re: (Score:2)
So, call me crazy, but didn't they do exactly what they are supposed to do? Also, AWS quite clearly states that any given AZ *might* fail. Hence, if you want any sort of high-availability, you replicate across different AZs.
For whatever reason, many of AWS's biggest flameouts have happened at the Virginia datacenter.
Between bad weather, rickety power infrastructure, bad hardware components, poorly configured software/hardware, etc etc etc
It's like setting up your data center in the Bermuda Triangle.
cloud is convenient, not reliable (Score:3)
As a cloud customer, reliability (currently at least) is up to you. If you want the extra reliability of running instances in multiple availability zones then it's up to you to pay for it.
The point of the cloud as it stands currently is not that it's cheap or reliable, but that it's easy to scale up/down with demand.
Re: (Score:2)
The first rule of diversification: don't put your eggs into correlating baskets.
In this context it means that if your primary is on AWS, then your secondary must be on Rackspace or whatever - NOT other AWS.
Re:Say what you will (Score:4, Insightful)
But that's the problem. *THEY* (i.e., AWS or whoever) are supposed to take care of all that stuff. They're supposed to worry about "uptime" and fixing things when they break and having redundant systems that kick in when something breaks so that there's no loss of service. That's the whole point of putting stuff in the "cloud".
Boy have you been fed a line. Read the SLA. If it's not in there; then you don't get it.
If you think the cloud provider is clustering your instance and giving you HA; then AWS is not for you.
Amazon provides availability zones you can provision separate instances storage and networks in. If your application cannot survive the failure of an instance and the failure of an entire availability zone, then you don't have HA, and Amazon won't give it to you -- your app may be inappropriate for AWS, if HA is required.
Re: (Score:2)
Amazon provides availability zones you can provision separate instances storage and networks in. If your application cannot survive the failure of an instance and the failure of an entire availability zone, then you don't have HA, and Amazon won't give it to you...
Not for free, they won't, but you most certainly can buy an HA solution from them.
Re: (Score:2)
Not always. Sometimes they provide *unavailability* zones.
That's just a play on words: they are availability zones. Just because you don't like how frequently they have a zone-wide outage doesn't change the name.
Another word for their availability zones is fault domain or failure domain.
If you don't want two things to fail together, you run them in a different region, using only resources in separate availability zones in separate regions.
Re:Say what you will (Score:4, Informative)
But that's the problem. *THEY* (i.e., AWS or whoever) are supposed to take care of all that stuff. They're supposed to worry about "uptime" and fixing things when they break and having redundant systems that kick in when something breaks so that there's no loss of service. That's the whole point of putting stuff in the "cloud".
Then either you're incredibly naive or you've never looked at what you get with most cloud providers.
Those £15/month virtual servers? You don't get any redundancy on those. If you're lucky, the provider will move it to a new physical host if the one it's living on breaks down, but they won't make any guarantees regarding how quickly that will happen or how automated and transparent that process is.
IME, the pile-it-high, sell-it-cheap brigade are punting exactly this. It's a whole bunch of physical boxes running something like Xen with a web-based front end but none of the work necessary to make it truly highly available has been carried out.
You want true high availability in the cloud - where even an entire datacentre going dark won't affect you? Well, then you have two choices:
- Architect your own. This means you will need several cheap virtual servers and you'll have to write your own software that accounts for all the various failure modes. Yes, this is difficult. Yes, this means you can't just fire up an Ubuntu image with Apache preinstalled on AWS and forget about it. Yes, this means it's a hell of a lot more expensive because suddenly you need to pay for lots of virtual servers rather than just one or two and you need to put a hell of a lot more work into the development process. But that was a choice you made when you went for the cheap option. Oh, you thought that because they used the word "cloud" in their marketing, that meant they'd already done all that for you? Ah.... no. Sorry.
- Contract it out to a company that has already built all this at the virtualisation level so you don't need to worry about it at the OS level. They operate a highly-available infrastructure with redundant everything and guarantees that even if something does fail, the redundancy will kick in automatically and you'll see no downtime. There are companies that offer this, but you might want to sit down with a strong drink before you look at their pricing structure. Clue: It's a hell of a lot more than £15/month for a basic virtual server.
Re: (Score:2)
But that's the problem. *THEY* (i.e., AWS or whoever) are supposed to take care of all that stuff.
They are, if you pay for "all that stuff". If you only pay for some of that stuff, you don't get HA.
They're supposed to worry about "uptime" and fixing things when they break and having redundant systems that kick in when something breaks so that there's no loss of service.
Again, AWS does "worry" about such things, and yes, they do have systems in place so that there's no loss of service - for customers who have purchased that level of service.
Re: (Score:2, Insightful)
No they didn't lie. You can set things up that way-simply set up your servers in multiple data centers(AWS availability zones) and load balance between them. It's foolish to just throw things up in the cloud and think magically I won't ever have to worry about downtime ever again.
But that was one of the big promises of "the cloud": that you'd never have to worry about the nitty-gritty of network administration again, your provider would handle all that for you. If that isn't the case, then you gain nothin
Re:Say what you will (Score:4, Interesting)
No they didn't lie. You can set things up that way-simply set up your servers in multiple data centers(AWS availability zones) and load balance between them. It's foolish to just throw things up in the cloud and think magically I won't ever have to worry about downtime ever again.
But that was one of the big promises of "the cloud": that you'd never have to worry about the nitty-gritty of network administration again, your provider would handle all that for you.
There are many different flavors of "cloud" computing - if you throw your app at a cloud provider and blindly expect them to make it highly available, then you'll get what you deserve. There is no end of cloud solution providers that will be happy to help you architect your app for whatever level of redundancy you want. But it's not going to be free.
Amazon does let you get rid of your network admin and concentrate on managing the servers. No need to worry about BGP, buying bandwidth from multiple redundant providers, buying and administering your own firewalls, network switches, routers, etc.
But you still have to manage your servers. Amazon will help you with multi-AZ redundancy for things like MySQL.
If that isn't the case, then you gain nothing and might as well host the data yourself.
That's depends heavily on your use case. If you have a relatively small number of servers, or have large demand spikes, Amazon can be much more cost effective than hosting your own servers. If you have hundreds of servers and keep them busy all the time, you can probably save money by doing it yourself.
But if you have dozens of servers, then it's likely that you'll save money with Amazon over buying your own servers, network gear, a SAN, backup solution, hardware service contracts, etc.
But you have to architect your application properly. We have our core servers split across multiple AZ's with the database replicated across those AZ's. We don't trust our failover/failback scripts enough to make it automatic, so we have a simple web interface to let anyone on the tech team do the failover. The only impact we saw in this outage was higher latency and timeouts to some of our app servers, but our database was not in the affected zone, and Amazon's load balancer correctly routed traffic to the servers in the good AZ.
Additionally, we have a warm spare running in a different region - the servers are kept up to date with data, but they are running in smaller instance types than we need to run our app, do to a regional failover, we'd have to reboot them into larger instance types (our app startup scripts already tune memory parameters to take advantage of the greater amounts of RAM in the larger instances), then repoint DNS.
Re: (Score:2)
Outages with AWS and cloudy friends are becoming so common it's almost a non-story at this point.
Re: (Score:3)
One of the features of AWS was supposed to be the ability to reroute everything to a different datacenter if one goes down. I know I read that somewhere back when AWS was first starting up. You don't think they lied, do you?
That all works just fine - if you build your application to use it.
However, nobody does this, because when your coworker is working on a new feature he can show the boss when it comes time for bonuses, do you want to spend your time working on something that you can only show off when Netflix goes down? Oh, and the way that you know that your work did anything is because your coworker's new feature still works.
Reliability is a hard personal sell in IT, and that is why there is so little of it. Anybody who
Re: (Score:2)
That's not a feature, and I don't recall it ever being advertised as one. You get to decide which zone and which datacenter(datacenters contain multiple zones) your instances/data lives in. If you want to replicate those to other zones and datacenters, it would be a very good idea. Amazon can't automagically do that for you though, and does not advertise that it can.
Re: (Score:2)
Re: (Score:3)
Unfortunately, "should" is rarely "does". If a brower receives multiple IP addresses for a name, it doesn't try them in turn, it just tries one.
Re: Say what you will (Score:2, Informative)
Most modern browsers do, indeed, try the next address. It' s a browser feature, though, not an official standard.
Re:Say what you will (Score:5, Informative)
Assuming you mean traditional round-robin A records, the timeout(s) you still have to suffer through would kill your latency.
If your talking about DNS providers (disclaimer, I work for Dyn) with advanced features that detect a failover event occurring and will only serve healthy A records, then that is a different story.
Re: (Score:2)
Re: (Score:3)
Most clients either won't move on to a second IP at all or will only move on once the OS times out the first TCP connection. And OS TCP connection timeouts are long enough that most users won't put up with them for interactive services.
A better strategy is to put a DNS server in each datacenter, make the TTLs short and set things up to automatically remove records if a server goes offline. This works much better because DNS fallback timeouts are much shorter than TCP connection timeouts.
Re: (Score:2)
For what you are stating, DNS servers at both using anycast, so the one that's u
Re: (Score:3)
Short TTLs are good only for changes (i.e. TTL at 24 hours, then set it to 12 hours 24 hours before an IP migration, then 4 hours 12 hours before a migration, then 1 hour 4 hours before a migration, then 5 minutes 1 hour before a migration, then to 24 hours after the migration is done), obviously set in seconds
Which works fine for planned migrations, useless for unplanned ones.
The down one won't ever announce itselft
Sure once a DNS server goes down it will stop sending responses regardless of whether you used anycast or the traditional system of just listing multiple DNS server IPs but responses it has already sent will continue to be used by clients until their TTL expires.
To make anycast useful without low DNS TTLs you would need to anycast not just the DNS servers but the servers they point to as well. However anycasting TCP based services causes br
Re: (Score:2)
To make anycast useful without low DNS TTLs you would need to anycast not just the DNS servers but the servers they point to as well. However anycasting TCP based services causes broken connections if routing changes.
Huh? It seems either you don't know how it's used, or you think I don't know how it's used. You over-simplified your answer to the point of technical falseness. Anycast is a routing trick to turn a unicast into a multicast-analogue (that is multi-unicast). As it is tecnically a routing function, yes, routing changes can break it. But pointing that out is like saying "breaking your car can break your car". It's a tautology. Changing routing between the anycast server and the recipient devices doesn't
Re:Say what you will (Score:4, Interesting)
Supposedly the load balancer problem did not affect LBs that have backing hosts in two availability zones according to the article. The major question is... who runs everything in one availability zone? You're not supposed to do that for high availability sites.
And you can do it with AWS (Score:3)
Re: (Score:2)
Re: (Score:2)
So yes, making resilient architecture on top of AWS is possible and is not that hard. You'll definitely have to pay extra money for it, but much less if you tried to build it yourself.
Re: (Score:2)
Actually, regardless, money becomes an issue really fast anyway - few days ago Wired run a story that for many types of loads AWS does not make much financial sense anymore and people started to add two and two together. In other words - people are prepared to pay only so much (in a pinch a little bit extra) - ask a little bit more and they'll start to roll their own.
Re: (Score:3)
Re: (Score:2)
Now add cooling, power, generators, physical security, a SAN, a virtualization platform, and multiple failover sites.
Re: (Score:2)
Cooling? Everyone has that.
Power? Yeah, you pay for that anyway.
Generators? Not as expensive as you think. I lived on a farm in the middle of nowhere once, and we had our own water supply and back-up generators.
Physical security? If you think that you're less likely to enjoy security with servers on your own site than controlled by some random third party who won't let you onto their site, who won't let you audit any of their processes, and who is almost certainly happily giving over information on request
Re: (Score:2)
You look after 4 servers. Amazon looks after 100,000 times that.
If every server has a 1 in 100 chance of failing each year, you have to wait over 10 years to reach a 50% chance that a server has failed.
Amazon would have about 11 dying per day. Its amazing that their systems can handle 99% of those failures seamlessly.
(My math may be way out but you get the point)
Re: (Score:3)
You look after 4 servers. Amazon looks after 100,000 times that.
I thought there were no servers in the cloud, just people willing to take your money and piss on you.
Re:Lack of reliability (Score:4, Interesting)
But I thought the whole point of the cloud was that everything included redundancy, so a server, or a cable, or a whole datacentre could go down, and because of real time replication, nothing whatever would be missed.
Or am I just thinking of VAXclusters from, you know, the 1980s.
Re: (Score:2)
But I thought the whole point of the cloud was that everything included redundancy, so a server, or a cable, or a whole datacentre could go down, and because of real time replication, nothing whatever would be missed.
Or am I just thinking of VAXclusters from, you know, the 1980s.
Well, that (VAX clusters) or the suite of services that AWS offers that would have prevented such an outage from affecting those customers who chose to utilize them. If you haven't deployed your stuff in multiple availability zones, along with the pieces that are required to tie that together into an HA stack, you don't get "the cloud". Why, oh why, does every article about every outage affecting a single AWS zone fail to mention this?
Re: (Score:2)
How is it that AWS is less reliable than the
How is it that AWS is less reliable than amazon.com ?
Seems Amazon occasionally claims to use AWS - yet amazon.com doesn't seem to die as much.
Are the rest of us just not using it correctly?
Re: (Score:2)
Are the rest of us just not using it correctly?
Oddly, yes.
Look for your 'single points of failure', including the cloud you're hosting in. Are you certain the cloud service is giving you redundancy on network, power, cooling, compute capacity, storage, physical location? What's their DR approach, and is your use of their service going to benefit from it?
Why are you using only one cloud hosting service, if uptime's that important to you.
Re: (Score:3)
Are the rest of us just not using it correctly?
More than likely.
You need to deal with the possibility of both individual EC2 instances dying and a whole availability zone dying in your application architecture. Amazon provides tools to help with this like load balancing that can operate over multiple availability zones and database service that are replicated across multiple availability zones but you have to actually use those tools if you want to build a reliable application.
Re: (Score:2)
and a whole availability zone dying
Even doing that, you end up less stable than amazon.com : http://techblog.netflix.com/2011/07/netflix-simian-army.html [netflix.com]
Chaos Gorilla is similar to Chaos Monkey, but simulates an outage of an entire Amazon availability zone. We want to verify that our services automatically re-balance to the functional availability zones without user-visible impact or manual intervention.
Re: (Score:2)
It's a multi-CPU system?
Re: (Score:2)
Well I've wondered about that myself. Yes, US-East is not only less expensive than the other zones for quite a few services, there's also AWS pricing policies that make it more attractive to use US-East for a lot of things. For example, S3 ingress data transmission charges are waved in US-East but not in the other AZs so I have other AZs replicating to US-East for disaster purposes just because of that. If US-East goes down it doesn't matter because I have apps leveraging multi-AZ availability and when U