37signals To Delete AWS Account, Cutting Cloud Costs By Millions (theregister.com) 65
Software firm 37signals is completing its migration from AWS to on-premises infrastructure, expecting to save $1.3 million annually on storage costs alone. CTO David Heinemeier Hansson announced the company has begun migrating 18 petabytes of data from Amazon S3 to Pure Storage arrays costing $1.5 million upfront but only $200,000 yearly to operate.
AWS waived $250,000 in data egress fees for the transition, which will allow 37signals to completely delete its AWS account this summer. The company has already slashed $2 million in annual costs after replacing cloud compute with $700,000 worth of Dell servers in 2024. "Cloud can be a good choice in certain circumstances, but the industry pulled a fast one convincing everyone it's the only way," wrote Hansson, who began the repatriation effort in 2022 after discovering their annual AWS bill exceeded $3.2 million.
AWS waived $250,000 in data egress fees for the transition, which will allow 37signals to completely delete its AWS account this summer. The company has already slashed $2 million in annual costs after replacing cloud compute with $700,000 worth of Dell servers in 2024. "Cloud can be a good choice in certain circumstances, but the industry pulled a fast one convincing everyone it's the only way," wrote Hansson, who began the repatriation effort in 2022 after discovering their annual AWS bill exceeded $3.2 million.
Waive the egress fee (Score:2)
Why would Amazon waive the egrees fee? I don't see any upside to them to do that.
Re: (Score:1)
Amazon almost always waves the S3 data transfer fee's as there stupidly high from the start and they already raped you on daily storage fee's anyway which is the real bread & butter money maker.
Re: (Score:1)
A few (perhaps all) of the big cloud providers do this as of maybe a couple years ago.
There is no monetary upside, it's just "good will" against the argument that picking a cloud provider effectively locks you in due to the egress fees.
Lock-in does not look good to competition regulators, so they avoid this by waiving such fees.
Re:Waive the egress fee (Score:5, Interesting)
Full disclosure: I'm an AWS employee, although not directly involved in any customer deals.
Waiving the egress fees of an unwilling customers frees up more capacity more quickly for eager customers. Demand still far outstrips the offer, so that storage won't be sitting idle.
Re: (Score:2)
Since you as a customer owns the data it could also mean costly court cases if the data isn't released so it's easier to just waive the fees.
Re: (Score:2)
It is pretty much standard customer service.
They would rather waive a fee and have you remember that they were good to you (even if the service was out of your price range) than have you leave mad. You might come back to them again.
Re: (Score:2)
As opposed to, for example, T-Mobile, who made some extra cash by continuing to bill me after I cancelled, but assured that I would never return even if they were the last cell provider on earth.
Re: (Score:2)
If Amazon charges outrageously high fees for egress, clients won't even try because "if we don't like Amazon we have to pay too much to leave."
Re:Waive the egress fee (Score:4, Informative)
To be clear, AWS egress fees are fees charged for any movement of data out of AWS, to the Internet, to another cloud provider, or to your own on-premise data store (or between different AWS regions or storage tiers). It doesn't mean a customer is leaving AWS. For this company, it means AWS waived the cost of moving their 18 petabytes of data from Amazon S3. Moderately opaque description [amazon.com].
Oh, how about that? Waiving egress fees for customers leaving AWS is now a policy [amazon.com] as of March, 2024: "starting today, we’re waiving data transfer out to the internet (DTO) charges when you want to move outside of AWS. ... The waiver on data transfer out to the internet charges also follows the direction set by the European Data Act."
So they aren't being exceptionally nice to 37signals. Learn something new every day, so now I've met my quota.
Re: (Score:2)
Why would Amazon waive the egrees fee? I don't see any upside to them to do that.
To allay EU regulatory investigations into the anti-competitive result of charging unreasonably high and excessive egress fees
which "locks customers in" from moving to a different cloud provider.
See: https://www.theregister.com/20... [theregister.com]
Re: (Score:2)
There is also a known market dynamic where making it easy to get out of an arrangement, makes it more attractive to get into the arrangement. So with this move, Amazon understands that getting into AWS looks a little better. It's a smart move.
They're weirdly vocal about this (Score:1)
They're saving $2 million a year which is.... 10 engineers' salary? Maybe 15 with juniors. How many engineers do they have? 200? I guess this looks good on paper but It's not outrageously big savings. Probably the biggest outcome of this transition is that they've fully quantified what their system really is and that allows them to consolidate various features since literally everything in their business stack is accounted for now, and you get a bunch of efficiencies by dropping dead code/zombie services.
Re: They're weirdly vocal about this (Score:2)
Re: (Score:2)
This. Playing with their stats to make it look like leadership is making positive changes. This only needs to last until they sell, go public, the C levels move on, or some combination of those things.
Re: (Score:2)
I mean, if those numbers are accurate, it seems like more than just "playing with their stats". $2.2 mil in CapEx in a year avoids $3 mil in OpEx? For this sort of stuff, you generally would expect such a capital expense to be about 3 times higher than the operational expense, with the benefit of getting more years out of it instead of incurring it every single year. If CapEx paid off within the same year, that's a really substantive benefit.
I also find it credible, as I've dealt with businesses so enamo
Re: (Score:3)
Re: (Score:2)
They almost certainly increased staffing, you need capacity planning, ordering testing installing and then headcount to monitor and repair, plus engineering resources to manage all the day to day infra operations. Part of the aws pricing is paying for the headcount on their side to manage uptime and reliability.
Re: (Score:2)
Cloud is not just someone else's computer. It's also someone else's BUSINESS OPERATIONS
Re: (Score:2)
Yuuuuuup. Somewhere in their org chart they've added a VP of datacenter operations, at least one middle manager and 3-5 techs and probably an SWE to build/maintain their datacenter tooling. Before they just had to call some APIs, now they are the API
Re: (Score:2)
And people were all weirdly vocal about going to the cloud as well.
It's a big shift, between capex and opex. There are a class of investors that like capex, particularly for longer lived companies, and others that love opex, particularly if there's a large chance the company goes bust before it could get the benefit of big upfront capex.
The "everything should be opex" crowd has been absurdly the standard voice in the market, to the point where companies are encouraged to spend more opex in a year than it w
Re: (Score:2)
Life was better... (Score:1)
both a lack of accounting and accountability (Score:5, Insightful)
"...but the industry pulled a fast one convincing everyone it's the only way..." and "...discovering their annual AWS bill exceeded $3.2 million..."
Part of this guy's job is to not be flimflammed by overstated marketing claims. Did he not run the numbers beforehand?. And was his eye NOT on the costs and had to be 'discovered' as if it were hiding?
So now the same guy who was so easily flimflammed before is now claiming wisdom in the other direction. Apparently self-awareness isn't a requirement to be a CTO.
As a mid-manager grunt, would my job be in jeopardy if I was responsible for losing the company millions? I'd probably also be hoping to blame the 'industry' instead of taking accountability for a bad choice.
Re:both a lack of accounting and accountability (Score:5, Informative)
The cost increase is generally justified as an early business expense as 1) aws services are generally reliable, secure by default, and excellent uptime 2) cheaper than hiring a dedicated operations person, your developers can just cobble together something and go 3) you don't need to do capacity planning. database going underwater? convert it to one twice as powerful in about 15 minutes with zero downtime! magic
Capacity planning has to be done quarterly, you need good metrics to do that, then you have to order, assemble, test the hardware which for a one-off thing can be quick but as sustained business operations is a 2-4 week project from the initial request to having it serving production traffic. once growth levels off DIY datacenter stuff can make sense, but early stage with less than 50 employees the AWS premium makes a lot of sense. transitioning to dedicated hardware is also an indicator that growth has leveled off or become extremely predictable.
Re: (Score:2)
Thanks for the thoughtful reply. Everything you said is right. But from my reading, their repatriation is not presented as a normal part of business growth, but as industry tomfoolery that had him convinced of something else. That also might be part journalistic spin, not sure. And hell no, I didn't RTFA.
I know my questions might have seemed rhetorical, but they are the exact type of questions I'd be asking if I were on their board or the CEO. I'd also like to ask him what he'd do to a regular employee
Re: (Score:2)
Reading the headline carefully, they are saving millions in cloud costs. How much are their salary, electricity, infrastructure, equipment bills going to increase to manage the hardware that they now own?
Re: (Score:2)
Generally private cloud vs aws savings are in the realm of ~20% for an operation their size. They almost certainly picked up some headcount to manage these which offsets a lot of the savings but not all of it.
Re:both a lack of accounting and accountability (Score:5, Interesting)
This is the company that runs Basecamp. Their CEO writes for Inc. and this is all part of more publicity of them. They're not going to include any of the other costs because it doesn't have a story to tell then. As you pointed out, there's much more than just buying the hardware. There's managing it, electricity, backup, replacing drives, and much more. They didn't factor that in when making claims about savings in a number of articles on their change. It's all just more marketing for them to get more recognition and customers.
Frankly, if I was a customer I'd be very concerned. They don't have the expertise to run such a storage program. This is their first go at it. And all customer data depends on it.
This move is 100% driven by finance. All they talked about was savings. Clearly they care far more about the bottom line than doing what's best for their customers. Never once did they mention the benefit or potential issues this could cause their customers.
Re: (Score:2)
Re: (Score:2)
Funny, because I have mod points right now.
Re: (Score:2)
"Clearly they care far more about the bottom line than doing what's best for their customers."
Staying in business is good for customers.
The savings are just the premium for AWS (Score:2)
I don't have the other article in front of me, but these savings are just the premium for running on AWS versus doing it "in house". There's only so much the economies of scale can get you when it comes to buying and running server hardware. AWS must cost more because AWS must turn a profit. One thing to note, though, is that premium to run on AWS gets you a whole lot of convenience that are often times worth the bill, especially when your workload doesn't need the equivalent of a full physical machine.
The
Re: (Score:2)
Living in the South Americas, the service levels you can expect here with the tech companies in the cloud is...lesser than you in your post profess. The son of my boss started a career with a local cloud company. In this region is a large set of waterfalls (that dwarfs the Niagra waterfalls). Cloud companies are popping up near that location for the electricity that can be generated in this location. So after 3 years with the locals, he applied for a similar architect job with the new international cloud pr
Re: (Score:2)
So this is probably because of plummeting wages (Score:2)
So when they ran the numbers a year or two ago then they probably would have
Re: (Score:2)
This appears in the story: 37signals spent $1.5 million on 18 petabytes worth of Pure Storage kit that Hansson wrote will cost less than $200,000 a year to operate. It's the only thing they mention they are moving "in-house".
18 petabytes will take maybe 200 drives plus (around USD$130k) plus other bits and bobs like SSD caches and an InfiniBand, network so at $1.5M it doesn't sound like they are scrimping. They also don't specify what "in-house" means, but given they are moving from S3 it might might co-l
To cloud or not to cloud. (Score:3)
One downside (Score:2)
It all seems really nice on paper but how would you like to be the poor bastard in charge of keeping that many Dell servers operational.
Maybe Dell servers are built by different people in that company but my wife has to use Dell laptops from work and they've so far had to be replaced about twice a year.
Re: (Score:3)
The main issue with maintaining physical servers is datacenter costs and making a few unlucky underpaid employees keep the whole thing running 24/7. We used to have datacenter engineering teams, which got chopped down to a 5-person team, then down to 2 people. We had data centers in 5 different countries. It's hard to manage relationships with 5 different data centers when the company doesn't want to hire a team in each country.
We migrated to AWS and spent a lot of money, then we started figuring out how to
Re:One downside (Score:4, Informative)
Re: (Score:2)
Good to know (Score:1)
Dell consumer products and dell server's are very different in construction and support.
That's good to hear as I really hoped they would be more careful with server gear. I just have no info in that regard to know.
Re: (Score:2)
I've dealt with sites managing tens of thousands of hosts. It's surprisingly not *that* difficult. It's a niche community but they have the tooling to do it.
It would be *excruciating* to wrangle that many laptops with laptop style usage, but not so much for servers. Besides improved manageability, there's also the reality that no one cares too much about arbitrary box X in a sea of boxes, so no one's particularly bothered if you batch up repair actions to do once every couple of weeks or maybe once a mon
Re: (Score:2)
had to be replaced about twice a year.
That's really hard to believe. Six month lifecycles? There's just no way that is true.
For Real (Score:1)
That's really hard to believe. Six month lifecycles?
Most of them were failures of the power connector on the laptop (the power supply was fine). She does have to bring the laptops into work a few times a week so they gut plugged and unplugged, but at most once or twice a day, some days just left plugged in and she's not rough on laptops otherwise (the Mac I gave her for personal use lasted five years or so without any failures).
A few were trackpad failures. Maybe you could chalk up some of those to reall
How Many Times (Score:2)
Re: (Score:2)
About as much as we needed to cover the story of irrelevant companies migrating to cloud hosting?
Fixed versus variable model .. (Score:2)
Duh! (Score:1)
18 Petabytes? (Score:2)
Anyone else wondering what are in these 18 Petabytes?
Is it Basecamp's HEY email service? ...etc.
Other than that, it is project management software, chat,
Can't imagine how those latter parts add up to 18 PB.
Re: (Score:2)
18PB isn't really that much when talking storage these days. I mean it's large, but not crazy big. There's a price/size level where using Cloud is cheaper than running it on-prem because you get the features of enterprise storage without the CapEx, but probably somewhere around 5PB (that number is changing almost daily it seems) it's way cheaper to run on-prem. One or two good storage engineers could handle this easily.
Hahahaha (Score:2)
Ruby... (Score:2)
If they wrote their code in a language that doesn't burn CPU like it's a free limitless resource their costs would have been far lower.
What happens when you need to refresh hardware? (Score:3)
OK, what happens when you need to refresh hardware?
Sunk cost is sunk cost. The beauty of cloud services is that depreciation cost is absorbed over time and not all at once every three to five years. Systems just keep working and you don't have to hire a gang of thugs to replace your hardware, let alone pay them to do break/fix on your hardware. I've been there and it's not cheap.
I'm jaded after having to deal with so much 7+-year-old hardware way past its end-of-life at the data center with NO budget for refresh or even break/fix maintenance.
Re: (Score:2)
People in companies that run servers this long on-prem or co-location, do you really think they behave better in a cloud environment?
Sure, they wouldn't be able to run too old hardware anymore, but in my experience such persons find other ways to make a similar mess of cloud services too.
Not denying you your jadedness, but it wasn't caused by the old hardware. Well, not really, if you think a bit more about it. Unintentional or otherwise incompetence will make a mess of anything it touches. Cloud is no diff
Re: (Score:2)
I don't follow. The underlying hardware is refreshed regularly and without concerning me (the customer) and at a marginal extra cost to me.
That's a win-win.
I hate dealing with 7-year-old computer and networking hardware anymore.
Self hosting isn't for everybody (Score:2)
The small 10-person company I work for was paying a lot of money for renting a few VMs from a company in Arizona. Several thousand dollars every month. It was an awful deal. Any of the big cloud providers would have been cheaper. However, the software engineers were not really infrastructure people and had limited interest in learning anything more than what they learned in college about computer programming. So, for them, this solution was already stretching the limits of their knowledge and understanding
And so the pendulum swings back the other way... (Score:2)
Let's hope 37Signals provides good jobs with good pay for their infrastructure and broadband staff and doesn't burn them out with a 2-week pager rotation just to "save money", or force them into a crappy, damaging work environment because some middle manager decided to treat his people "on the cheap" while taking better care of the machines.