Signal Chief Explains Why the Encrypted Messenger Relies on AWS (theverge.com) 61
An anonymous reader shares a report: After last week's major AWS outage took Signal along with it, Elon Musk was quick to criticize the encrypted messaging app's reliance on big tech. But Signal president Meredith Whittaker argues that the company didn't have any other choice but to use AWS or another major cloud provider.
"The problem here is not that Signal 'chose' to run on AWS," Whittaker writes in a series of posts on Bluesky. "The problem is the concentration of power in the infrastructure space that means there isn't really another choice: the entire stack, practically speaking, is owned by 3-4 players."
In the thread, Whittaker says the number of people who didn't realize Signal uses AWS is "concerning," as it indicates they aren't aware of just how concentrated the cloud infrastructure industry is. "The question isn't 'why does Signal use AWS?'" Whittaker writes. "It's to look at the infrastructural requirements of any global, real-time, mass comms platform and ask how it is that we got to a place where there's no realistic alternative to AWS and the other hyperscalers."
"The problem here is not that Signal 'chose' to run on AWS," Whittaker writes in a series of posts on Bluesky. "The problem is the concentration of power in the infrastructure space that means there isn't really another choice: the entire stack, practically speaking, is owned by 3-4 players."
In the thread, Whittaker says the number of people who didn't realize Signal uses AWS is "concerning," as it indicates they aren't aware of just how concentrated the cloud infrastructure industry is. "The question isn't 'why does Signal use AWS?'" Whittaker writes. "It's to look at the infrastructural requirements of any global, real-time, mass comms platform and ask how it is that we got to a place where there's no realistic alternative to AWS and the other hyperscalers."
HyperScalers (Score:3)
We are no longer just "web scale," we are now at "hyper scale." That sounds so, like, high tech and stuff.
Re:HyperScalers (Score:5, Funny)
We are no longer just "web scale," we are now at "hyper scale." That sounds so, like, high tech and stuff.
When do we get to ludicrous scale?
Re: (Score:3)
obligatory plaid joke.
Scary (Score:2)
Re: (Score:3)
I wonder if it would make sense for a company like Signal to have servers in both AWS and Azure. Thus lowering the chances of a full crash. At least have a degraded speed mode or something.
But at some scale hosting your own servers becomes cheaper. And you use AWS as your backup.
Re: (Score:2)
Multi-cloud increases your points of failure. Even multi-region within AWS adds complexity for data synchronization, and complicates the restoration of service after a single-region failure.
Re: Scary (Score:2)
That sounds like a job for ipv6 anycast.
Re:Scary (Score:5, Insightful)
All redundancy increases points of failure and complicated fixing things. That doesn't make redundancy a bad idea.
Re: (Score:2)
Multicloud would be a good option but what's the point? Your whole system can go down but you can point the finger at AWS and nobody even questions it. We saw that with CrowdStrike too.
It's sort of similar to having your personal information released in a breach these days. Nobody cares anymore because everyone already has your information. Once it's expected, it no longer matters.
Re: (Score:2)
+1 insightful
Learn from kiwifarms (Score:3)
Kiwifarms seems to have figured out how to solve this problem pretty well. No cloud providers will host them due entirely to public pressure, so they've created a distributed hosting system with it's own ddos mitigation. And it seems to hold up well as they get ddos'd quite a bit, and any time one server goes down, there's another somewhere else on some other smaller provider to take it's place. This is exactly why I find that site so interesting. I don't care for the content, but they've proven that even in this day, you don't have to rely on any big infrastructure provider.
However, it isn't a turnkey solution, unlike say AWS, who owns practically the whole stack. A lot of work and maintenance has to be done. I bet somebody could come up with a software framework of sorts that makes managing multiple providers on your own easier, and more scalable than what kiwifarms has done. Kiwifarms didn't even try to do it until they had to. And right now, nobody else has to.
But do companies like signal have the will to?
Re:Learn from kiwifarms (Score:4, Informative)
It is possible, today, to multi-home. Critical services do that (or should). It's not even *that* hard, although you have to be competent with all your providers.
I'll oversimplify a bit, but
Note that all of this assumes one of the following: either your providers play nice with each other, or you go bare metal in some places to setup redundancy, replication, etc.. Also note that this will require competent sysadmins to at the very least bridge the gap between multiple providers. A business could be built on top of these ideas, to provide services that are "transparently" served by at least two big, independent providers. But all of this costs a lot.
About Signal they failed when AWS failed. Baring any active attempt at hindering Signal, this is a relatively rare event (although it happens more and more these days). Increasing infrastructure costs indefinitely to handle a very sporadic downtime is not really an obvious solution. Keep in mind that as long as it's not the whole AWS zone that goes under, AWS is quite good at keeping services available in case of minor events (other providers too, I suppose).
I have no idea where Signal gets his money, but wherever that is, I assume that "increasing infrastructure cost by 40% to alleviate two hours of downtime a year" would not look too good.
Re: (Score:1)
Signal sells on their security. Being on AWS makes every message far-more-easily available to governments and Amazon's interests. Even if the apps are secure when the network is not, the apps can receive a sneaky update at any time when the files are not served by the owner.
Re: Learn from kiwifarms (Score:2)
What I have in mind is more akin to a kubernetes-like framework, combined with ipv6 anycast IPs (so DNS is less important, especially if your app manages it's own certificates, does certificate pinning, etc.) If such a thing existed, I think it could prove to be pretty reliable.
We already know that it can be done with competent sysadmins, because somebody is already doing it.
The part that software can't solve is setting up contracts with the providers, and they're all doing their own thing. On the other han
Re: (Score:2)
A
That's basically how all of our multi-PoP HA works.
As someone mentioned, synchronization gets more difficult as the system gets more complex, but frankly it isn't bad, and there are *l
Re: Learn from kiwifarms (Score:2)
The problem with v4 anycast is the requirements for it (at least, for a public IP) are beyond the reach of what most people can reasonably do, even if they know what they're doing.
If the entire internet hates you, or you just plain don't have the resources, and you're susceptible to ISPs that suddenly drop you and/or go out of business, I think having more than one v4 anycast subnet might be hard to maintain uptime with.
I've never had to though, so I'm not sure. I tend to think that having multiple would at
Re: (Score:2)
The problem with v4 anycast is the requirements for it (at least, for a public IP) are beyond the reach of what most people can reasonably do, even if they know what they're doing.
If the entire internet hates you, or you just plain don't have the resources, and you're susceptible to ISPs that suddenly drop you and/or go out of business, I think having more than one v4 anycast subnet might be hard to maintain uptime with.
I've never had to though, so I'm not sure. I tend to think that having multiple would at least give you redundancy without having to rely too much on DNS, especially as in the case with kiwifarms where one of their registrars just decided to keep the domain after dropping them (and it's taking them on the order of years so far to get it back via the courts.) Though I suppose for an app, using multiple domains might make that mostly transparent
The crazy thing is the internet was designed to route around politicians and hardware failures. It just doesn't work too well when either the internet hates you or your cloud provider goes down. Signal I think is exactly the kind of app that needs to have distributed services, regardless of what the internet thinks of it.
Wait- is there a way to anycast an IPv6 subnet without arranging for transit from someone? Shouldn't be any difference between getting your hands on an IPv4 allocation and an IPv6 allocation and then setting up transit (again- other than price, which is a major hurdle since IPv4 addresses are worth their weight in gold)
If the entire internet hates you...
I think that also comes down to price, really. The smallest IPv6 subnet you can reasonably announce on the public internet is still going to be very cheap, and come in a larger network of man
Re: (Score:2)
In fairness, Signal has something on the order of 70 million active users, and in contrast this is the first time I've ever heard of Kiwifarms.
Re: (Score:3)
Kiwifarms is a tiny website though. Signal handles a lot of messages, a lot of push notifications. An even bigger issue is that Signal needs to be available as widely as possible, without being blocked.
Tor uses Microsoft Azure to get around blocking in some regions. Even governments can't really block what look like normal HTTPS connections to Azure cloud, without breaking a lot of stuff. The same goes of AWS.
Re: (Score:2)
Tor uses Microsoft Azure to get around blocking in some regions. Even governments can't really block what look like normal HTTPS connections to Azure cloud, without breaking a lot of stuff. The same goes of AWS.
Blackhats like to host proxies for their traffic in Azure and AWS for the same reason.
Monopolism (Score:2)
Re:Monopolism (Score:5, Insightful)
Re:Monopolism [solution approach] (Score:4, Interesting)
What if we taxed monopoly profits in a progressive way? If you are getting profit from a monopoly, then your tax rate goes up, with the revenue used for regulating the monopoly and for researching ways to break the monopoly. Various ways to detect monopoly situations, but the three I favor right now are (1) checking to see if customers have real choice, (2) studying complaints from wannabe competitors who can't get into the market, and (3) looking at whether the people who work in the field have alternative employment opportunities.
But my truly subversive thought is that this would motivate companies to break their own monopolies to increase their retained earnings. Divide the company into competing entities. The evil that is Microsoft is an easy example. Imagine that Microsoft split itself into two or three daughter companies, each starting with a copy of the source code and an equal share of the corporate resources. Then let them go at it hammer and tongs. (Plus none of them needs to be too big to fail and all of them can and should be subject to higher liability for mistakes and incompetence...)
Solution-oriented thinking? I must be begging for the Funny mod.
Re: (Score:3)
Companies have been broken up in the past, with good results. A company might split itself to get around antitrust laws, which is fine if it results in actual competition.
Re: (Score:2)
ACK and concurrence with your extensions.
Re: Monopolism (Score:4, Interesting)
Re: (Score:2)
Re: (Score:2)
That's kind of what that (tired and overused term) "late stage capitalism" is: monopolies, oligopolies, and lack of competition.
So "late stage capitalism" started in the 1800's? In theory we broke up those monopolies in the United States around 1900 with anti-trust laws. In practice, the modern Supreme Court has gutted those laws. That is not surprising when the entire court attended one of two law schools, both of which focus on training lawyers to serve large corporations.
Re: (Score:2)
Yes, the late 1800s-1920s was peak late stage capitalism, but the threat of communism made it change its ways and play nice for like 30-40 years, but then people were successfully indoctrinated into letting capitalism run amok by successive waves of red scares and the USSR collapsed, and now here we are again. Every time capitalism survives a brush with its late-stage phase, it means we will suffer through another one, but with more automation, surveillance and means of control.
Re:Monopolism (Score:4)
Part of the government's job is to protect the health of the economy by breaking up monopolies and enforcing anti trust law. This is necessary in order to ensure that there remains opportunity for competition.
It isn't an easy thing to do, of course, especially when those with the most political power are the very monopolies and cartel bosses being regulated, but it is a necessary element of a healthy capitalistic economy.
Our current government isn't doing a very good job of that. And so, predictably, here we are.
Elon should just shut up. (Score:4, Informative)
Bluesky uses AWS.
Signal uses AWS.
Telegram uses AWS.
Instagram used AWS, now Facebook.
TikTok uses AWS and Google.
Elon should go back to doing drugs and insulting Trump.
Re:Elon should just shut up. (Score:5, Funny)
Elon should go back to doing drugs and insulting Trump.
"go back to"?
roll your own (Score:4, Insightful)
How did these cloud providers get to be "hyperscale"? They didn't start hyperscale, they grew into hyperscale.
Signal too could buy its own hardware, and put it in datacenters that they own or lease. No one is dependent on cloud providers, everyone can run their own hardware. Cloud providers have done a good job of convincing people not to run their own, there is nothing like instant gratification (at whatever cost). I'd guess Signal could save money running their own hardware.
Re: (Score:2)
Yeah, but that takes effort.
Re: (Score:2)
The money they give to Amazon could pay for a lot of effort.
Re: (Score:2)
At that scale, anyone would save massive money if they had a basic emulation layer to make their private environment look like AWS. That's likely part of it - Amazon uses a lot of open source software but they tweak it and nothing is exactly the same. The problem is either having to go multicloud during the transition (paying double) or having to cut over, not to mention rewriting parts of your software to work with non-AWS infrastructure.
Re: (Score:2)
Arguments to never start with AWS. Signal didnt benefit from their provider's availability zones, maybe they don't care to be more robust but want to tell a different story.
This is why corporate mergers/acquisitions are bad (Score:2)
Too easy to work around (Score:1)
>A corporation should start life as it's own entity an be required to end as it's own entity. No mergers or buy outs by other corporations.
Such a rule would be too easy to work around: Instead of "selling yourself" as part of a merger, sell your assets, little by little, to the company you want to be absorbed by, and passing the profits on to your stockholders as a special dividend (sadly for the stockholders, this will mean taxes on dividends). When you are down to nothing of value beyond your corpora
Why do you need cloud (Score:2)
In the thread, Whittaker says the number of people who didn't realize Signal uses AWS is "concerning," as it indicates they aren't aware of just how concentrated the cloud infrastructure industry is.
Why do you need cloud?
It's expensive. It puts an additional layer of potential failure. It makes you dependant and locks you in. The list just goes on and on. It might makes sense for a startup that doesn't want to commit to capital investment but even then there are less expensive options.
Re: (Score:2)
That's what MBA groupthink and software monoculture gets you.
Common logic among mid to C suites is "you'd be crazy to build what you can buy for cheaper"
CRAZY, I say! <slams table>
Until there is a worldwide outage, and you don't look so smart anymore as billions in sales or whatever is lost.
So until Management puts a value on self reliance, this will continue.
Sad. Pretty much ev
Re: (Score:3)
Until there is a worldwide outage, and you don't look so smart anymore
Except everyone just looked to Amazon for the blame. Not one company was heavily blamed for relying on AWS. But if they moved to private infrastructure every blip or outage would be their own fault with no finger to point.
This isn't the network we tried to build (Score:5, Insightful)
That was then. This is now. Back then, we thought about the long-term good of the network and its prospects for helping human society. Today everyone is thinking about next quarter's profits and nobody cares about the impact on people. And thus the original architecture of the network has been subsumed by a relatively small number of operations which in turn are in the critical path for hundreds of thousands of operations. The network is now -- ironically -- far more fragile than it was when we cobbled together connections between the ARPAnet, Usenet, CSNet, and BITnet with spare parts.
It's now entirely plausible that an adversary with a budget of under $1M US could cripple the country for days to weeks, disrupting air travel, commerce, utilities, telecommunications, etc. via simultaneous attacks on just a handful of operations. And the budget to do the same to other countries may be considerably smaller.
We can't fix this. We're old and dying off, and those few of us who are left are dismissed as out-of-touch and obsolete. It will be up to those of you who are much younger to reverse this by pushing -- hard -- to move things back to as distributed an architecture as possible. Good luck.
Re: (Score:3)
Engineers don't make decisions. Profits do.
Re: (Score:2)
Doesn't this make it a bit of a zero sum game, since everyone can essentially take out everyone else?
Re: (Score:2)
Re: (Score:2)
Thanks for clarifying!
The electrical grid is (was?) vulnerable to attack (Score:2)
>It's now entirely plausible that an adversary with a budget of under $1M US could cripple the country for days to weeks,
About 5-10 years ago I read that there were 2-3 dozen electrical substations that, if a key component at each one was taken out all at once, would cause major disruptions for large parts of the United States for years.
If you knew where these 2-3 dozen substations were, you might be able to take them all out at once for a budget of $10M (think: drone with bomb). Granted, $10M is far fr
Misleading premise (Score:3)
It's not like we lost something here. Before AWS and other cloud providers came along, it was simply impossible to stand up a global video conferencing network without a massive capex and ongoing maintenance costs if you weren't already a telco or backbone internet provider.
If all Signal wanted to do was exchange text coms with occasional images, they could still rent a handful of colo racks and be in pretty good shape.
Complaining about limited options to source a global multimedia streaming network solution is like complaining about limited options for contractors to put a satellite in orbit. Well yeah, it's a niche expensive service with huge startup costs. The fact that you can buy such a service at all is pretty amazing.
XMPP ? Jami ? Matrix ? Only on Hyperscalers ? (Score:2)
Why would you need AWS for exchanging just small text ? It is not true for emails. It is not true for messages either. Just use any federated text services. All of them have a very good support for encryption client to client.
bullkrap (Score:2)
big telcos worldwide (vodafone, orange, telefonica, destuche telekom) offer opestack clouds. ditto for other cloud actors (4th place biggest cloud onwards)
so signal could be running in 3 or four DIFFERENT openstack clouds so, if one fails, the others pick up
is just lazziness and herd mentality...
I saw this thru a link sent in Session (Score:1)
Delta Chat (Score:1)
One of the things myself and mates are testing is Delta Chat (https://delta.chat) over chatmail relays - it works surprisingly well, and the new design of chatmail relay onboarding is pretty darn easy. Using it is just fine (the UI is dated, could use a designer); I was worried about battery use but it seems to do a really good job, down at the 2% level with regular use since it's not using push tech, instead having to poll.
Why not peer to peer? (Score:2)
Why not use peer-to-peer encrypted messaging solutions that don't need central severs?
https://jami.net/ [jami.net] is one such system (I have no connection to it, I just think its a good system)
The necessity of AWS and centralized systems. (Score:1)