HN Reader

When I was learning to program through a bootcamp I spun up an elastic beanstalk instance that was free but required a credit card to prove your identity. No problem that makes sense - it's an easy way to prove authentication as a bot can't spam a credit card (or else it would be financial fraud and most likely a felony).

Amazon then charged me one hundred thousand dollars as the server was hit by bot spam. I had them refund the bill (as in how am I going to pay it?) but to this day I've hated Amazon with a passion and if I ever had to use cloud computing I'd use anyone else for that very reason. The entire service with it's horrifically complicated click through dashboard (but you can get a certification! It's so complicated they invented a fake degree for it!) just to confuse the customer into losing money.

I still blame them for missing an opportunity to be good corporate citizens and fight bot spam by using credit cards as auth. But if I go to the grocery store I can use a credit card to swipe, insert, chip or palm read (this is now in fact a thing) to buy a cookie. As opposed to using financial technology for anything useful.

The assignment of blame for misconfigured cloud infra or DOS attacks is so interesting to me. There don't seem to be many principles at play, it's all fluid and contingent.

Customers demand frictionless tools for automatically spinning up a bunch of real-world hardware. If you put this in the hands of inexperienced people, they will mess up and end up with huge bills, and you take a reputational hit for demanding thousands of dollars from the little guy. If you decide to vet potential customers ahead of time to make sure they're not so incompetent, then you get a reputation as a gatekeeper with no respect for the little guy who's just trying to hustle and build.

I always enjoy playing at the boundaries in these thought experiments. If I run up a surprise $10k bill, how do we determine what I "really should owe" in some cosmic sense? Does it matter if I misconfigured something? What if my code was really bad, and I could have accomplished the same things with 10% of the spend?

Does it matter who the provider is, or should that not matter to the customer in terms of making things right? For example, do you get to demand payment on my $10k surprise bill because you are a small team selling me a PDF generation API, even if you would ask AWS to waive your own $10k mistake?

I thought this would be about the horrors of hosting/developing/debugging on “Serverless” but it’s about pricing over-runs. I scrolled aimlessly through the site ignoring most posts (bandwidth usage bills aren’t super interesting) but I did see this one:

https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-c...

About how you make unauth’d API calls to an s3 bucket you don’t own to run up the costs. That was a new one for me.

I don't understand why it should be called "serverless" when using cloud infrastructure. Fundamentally you're still creating software following a client-server model, and expecting a server to run somewhere so that your users' clients work.

To me, "serverless" is when the end user downloads the software, and thereafter does not require an Internet connection to use it. Or at the very least, if the software uses an Internet connection, it's not to send data to a specific place, under the developer's control, for the purpose of making the software system function as advertised.

The real serverless horror isn't the occasional mistake that leads to a single huge bill, it's the monthly creep. It's so easy to spin up a resource and leave it running. It's just a few bucks, right?

I worked for a small venture-funded "cloud-first" company and our AWS bill was a sawtooth waveform. Every month the bill would creep up by a thousand bucks or so, until it hit $20k at which point the COO would notice and then it would be all hands on deck until we got the bill under $10k or so. Rinse and repeat but over a few years I'm sure we wasted more money than many of the examples on serverlesshorrors.com, just a few $k at a time instead of one lump.

> I had cloudflare in front of my stuff. Hacker found an uncached object and hit it 100M+ times. I stopped that and then they found my origin bucket and hit that directly.

Pardon my ignorance, but isn’t that something that can happen to anyone? Uncached objects are not something as serious as leaving port 22 open with a weak password (or is it?). Also, aren’t S3 resources (like images) public so that anyone can hit them any times they want?

"Serverless" is a an Orwellian name for a server-based system!

I keep telling customers: "The cloud will scale to the size of your wallet."

They don't understand what I mean by that. That's okay, they'll learn!

Anyway, this kind of thing comes up regularly on Hacker News, so let's just short-circuit some of the conversations:

"You can set a budget!" -- that's just a warning.

"You should watch the billing data more closely!" -- it is delayed up to 48 hours or even longer on most cloud services. It is especially slow on the ones that tend to be hit the hardest during a DDoS, like CDN services.

"You can set up a lambda/function/trigger to stop your services" -- sure, for each individual service, separately, because the "stop" APIs are different, if they exist at all. Did I mention the 48 hour delay?

"You can get a refund!" -- sometimes, with no hard and fast rules about when this applies except for out of the goodness of some anonymous support person's heart.

"Lots of business services can have unlimited bills" -- not like this where buying what you thought was "an icecream cone" can turn into a firehouse of gelato costing $1,000 per minute because your kid cried and said he wanted more.

"It would be impossible for <cloud company> to put guardrails like that on their services!" -- they do exactly that, but only when it's their money at risk. When they could have unlimited expenses with no upside, then suddenly, magically, they find a way. E.g.: See the Azure Visual Studio Subscriber accounts, which have actual hard limits.

"Why would you want your cloud provider to stop your business? What if you suddenly go viral! That's the last thing you'd want!" -- who said anything about a business? What if it's just training? What if your website is just marketing with a no "profit per view" in any direct sense?

Hetzner, 16TBx2 HDD, 1TBx2 SDD, 64GB RAM, 20TB free bandwidth, $70/month.

I used 1TB of traffic on a micro instance and it cost me $150 (iirc). Doesn't have to be this way.

After a quick check on Vercel stories, it seems all payments were discarded or mistakes in the first place.

Does it really happen to really have to pay such a bill? Do you need to tweet about it to be reimbursed?

It would help to round to the cent. With 3 digits to the right of the dot it's ambiguous whether it's a decimal point or a thousands separator, and the font and underline makes the comma vs dot distinction a bit unclear.

Putting any sort of pay per use product onto the open internet has always struck me as insane. Especially with scaling enabled.

At least stick a rate limited product in front of it to control the bleed. (And check whether the rate limit product is in itself pay per use...GCP looking at you)

I tried AWS serverless, figured out that it is impossible to test anything locally while you are forced to use AWS IAM role for serverless run which has access to everything.

That's just a problem waiting to happen while you are always running tests on production...

This is some good marketing for Coolify, which the author makes as an open source platform as a service. I prefer Dokploy these days though, since it seems to be less buggy, as Coolify seems to have such bugs due to being on PHP.

https://coolify.io/

https://dokploy.com/

last employer asked for an estimate to migrate to cloud.

it would be 2x more expensive and halve developer speed. also we would lose some internal metric systems honed over 20yr.

ceo told to go ahead anyway (turn out company was being sold to Apollo)

first thing we did was a way to bootstrap accounts into aws so we could have spend limits from day one.

can't imagine how companies miss that step.

Are there any protections these days at the cloud provider level?

Like setting a maximum budget for a certain service (EC2, Aurora?) because downtime is preferable to this?

These guys charge $550 for a measly terabyte of bandwidth?

If you get a dedi on a 10Gb/s guaranteed port and it works out to more than $3 / TB, you're probably getting scammed. How does "serverless" justify 150x that? Are people hosting some silly projects really dense enough to fall for that kind of pricing?

Just get a $10 VPS somewhere or throw stuff on GH pages. Your video game wiki/technical documentation/blog will be fine on there and - with some competent setup - still be ready for 10k concurrent users you'll never have.

For everyone complaining about no free tier that blocks you from being charged

https://aws.amazon.com/free/

Experience AWS for up to 6 months without cost or commitment

Receive up to $200 USD in credits

Includes free usage of select services

No charges incurred unless you switch to the Paid Plan

Workloads scale beyond credit thresholds

Access to all AWS services and features

I once found an official Microsoft example repo to deploy an LLM gateway on Azure with ALB. Glad I did the tedious work of estimating the costs before I hit the deploy button (had to go though many Biceps manifests for that). The setup would have cost me about 10k/month.

I remember at the beginning of the serverless hype how they said it was great because it automatically scaled as big as you need it. Given how sudden and massive these "scaling spikes" can be, I would much rather deal with a death-hugged VPS than a $100k bill.

Plus the VPS is just so much faster in most cases.

This is why when I contract for an early stage startup, I pose the question:

"What if your app went viral and you woke to a $20k cloud bill? $50k? $80k?"

If the answer is anything less than "Hell yeah, we'll throw it on a credit card and hit up investors with a growth chart" then I suggest a basic vps setup with a fixed cost that simply stops responding instead.

There is such a thing as getting killed by success and while it's possible to negotiate with AWS or Google to reduce a surprise bill, there's no guarantee and it's a lot to throw on a startup's already overwhelming plate.

The cloud made scaling easier in ways, but a simple vps is so wildly overpowered compared to 15 years ago, a lot of startups can go far with a handful of digitalocean droplets.

Does anyone heard a success stories from cloud usage?

Really, we(they) in the company decided to move in cloud everything from on-prem, it should save costs say them.

But, as result you anyway need DevOps, some complications with development, local environments and not only.

For not short career I faced some good examples, but it's more about unique situations, not a rule and a lot companies continue pay a lot for some small bunch of utility.

Maybe I'm wrong, but such topics about this hell heard a lot of timesand only on some conference: success stories (because they should say: success)

Don’t most of these services have config options to protect against doing this? I haven’t used most of these services but it running up a bill during traffic spikes but not going down seems like it’s working as intended?

Yeah I also left my website hosted on Google Cloud because costs popped from everywhere, and there is basically no built-in functionality to limit them. So I didn't really slept relaxed (I actually slept great, but I hope you get the point) knowing that a bug could cost me... who knows how much. Actually, as the website of OP says, for spending control you have budget notifications and with that you can disable the billing for all the project altogether through some API call or something, I don't remember exactly, that is all there is. But still it looks like this functionality is just not there.

This site is a bit dated. I remember in response to this Vercel added a way to pause your projects when hitting a spend limit. I enabled it for my account.

Still, it made me question why I'm not using a VPS.

Couple years ago I was charged in USD 4K on Google Cloud after trying recursive cloud functions.

I told them that was a mistake and they forgot the debit, they just asked to no do again.

I can't imagine hosting a small-time project on rented infrastructure without some kind of mechanism to terminate it once costs exceed a reasonable threshold.

Troy Hunt and HIBP is a good example in the other direction but Hunt has also been burned plenty of times by serverless.

https://www.troyhunt.com/closer-to-the-edge-hyperscaling-hav...

Andras (author of Serverless Horrors) knows what he’s talking about.

The amount of brainwashing that big cloud providers have done, is insane.

Seem likes there are mistakes that were made on behalf of the users. The attackers found these mistakes and took advantage of them. i don't think "severless" is the problem.

We are building bare metal for our workloads… I don’t care if cloud is supposed to be cheaper because it never is. You can get a decent small business firewall to handle 10gbit fiber for $600 from unifi these days. Just another reason I’m glad I moved out of the Bay Area and nyc to a midwestern town for my company. I have a basement and can do rad things in my house to grow my business.

There should also be a general category for "cloud horrors" for things that cost $50k/month to host that would be $1500/month on a bare metal provider like Datapacket or Hetzner.

I'm old enough to remember when cloud was pitched as a big cost saving move. I knew it was bullshit then. Told you so.

I guess I'm missing something, why is this 'serverless' horrors? If anything it seems to specifically be serverful horrors.

In my experience: Fuck serverless.

If we're building anything bigger than a random script that does a small unit of work, never go for serverless. A company I recently worked for went with Serverless claiming that it would be less maintenance and overhead.

It absolutely was the worst thing I've ever seen at work. Our application state belonged at different places, we had to deal with many workarounds for simple things like error monitoring, logging, caching etc. Since there was no specific instance running our production code there was no visibility into our actual app configuration in production as well. Small and trivial things that you do in a minute in a platform like Ruby on Rails or Django would take hours if not days to achieve within this so-called blistering serverless setup.

On top of it, we had to go with DB providers like NeonDb and suffer from a massive latency. Add cold starts on top of this and the entire thing was a massive shitshow. Our idiot of a PM kept insisting that we keep serverless despite having all these problems. It was so painful and stupid overall.

I've had this twice. Once with oracle, once with azure. They both charged me $2000-$5000 for simply opening and closing a database instance (used only for a single day to test a friend's open source project)

To be fair, support was excellent both times and they waived the bills after I explained the situation.

I got a large GCP logstorage bill as my applicatino is writing too much logs

look mom & dad, I am famous!

Title should really be "Cloud Hosting Horrors", not serverless per se.

At one time, I considered using Firebase as a backend, but then, I kept reading stories like these, and decided to roll my own. I'm fortunate to be able to do that.

It's kind of amazing, though. I keep getting pressure from the non-techs in my organization to "Migrate to the Cloud." When I ask "Why?" -crickets.

Industry jargon has a lot of power. Seems to suck the juice right out of people's brains (and the money right out of their wallets).

This is a weird take on an incredibly useful paradigm (serverless). One the one side, there are obviously precautions that all of these users could have taken to avoid these charges on the other hand its totally common to spin up a thing and forget about it or not do your due diligence. I totally feel for the people who have been hit with these chargers.

At the end of the day though the whole think feels like a carpenter shooting themselves in the foot with a nail gun then insisting that hammers are the only way to do things.

Maintaining your own containers or VMs is hard considering how much risk appetite you have for the issues at infra level. So, yeah, when you complain about the costs of serverless, you are just paying for your low risk appetite low cost of your IT management.

An alternative title might be "Failure to read the documentation horrors."

If you didn't sit down with the documentation, the pricing guide, and a calculator before you decided to build something then you share a significant portion of the fault.

How are all these cases of exorbitant surprise bills not prosecuted as fraud?

I read a lot of the posts at the little blog here and, uh, every single one sounds like a complete amateur making a cloud configuration mistake. I haven't found one that is the provider's fault or the fault of "serverless"

I would be embarrassed to put my name on these posts admitting I can't handle my configs while blaming everyone but myself.

Serverless isn't a horror, serverlesshorrors poster. You are the horror. You suck at architecting efficient & secure systems using this technology, you suck at handling cloud spend, and you suck at taking responsibility when your "bug" causes a 10,000x discrepancy between your expected cost and your actual bill.

Just because you don't understand it doesn't mean it sucks

Hahaha, this is awesome!

looking forward to the "LLM token horrors" version

I have a feeling I will be downvoted for this, but...

Have the people posting these horror stories never heard of billing alerts?

there should be some kind of insurance for bugs that introduce unusually expensive usage

I was also too careless with AWS when I was a beginner with no deployment experience and I am very lucky that I did not push a wrong button.

All these stories of bill forgiveness reminds me of survivorship bias. Does this happens to everyone that reaches out to support or just the ones that get enough traction on social media? I am pretty sure there is no official policy from AWS, GCP or Azure.

Related. Others?

Single day Firebase bill for $100k - https://news.ycombinator.com/item?id=43884892 - May 2025 (14 comments)

Serverless Horrors - https://news.ycombinator.com/item?id=39532754 - Feb 2024 (169 comments)