Tell HN: Heroku deleted my database with no warning

[fireworks]

Last December, Heroku nuked the database on one of my active projects. I was travelling at the end of the year and did not catch wind of this until I returned and saw messages about an issue with the app. Sure enough, I checked and noticed that the database was gone and detached on December 9th.

Before the hate comes out, yes I know Heroku deprecated free tiers. However, I did not understand this would affect my projects on paid dynos. The real issue here is that I never received a single email or notice of any kind to my email about this. From researching, it appears most people received SEVERAL notices about this. I did not think there was an issue with my setup because I received zero communication.

Upon reaching out, Heroku has told me that they cannot recover the database. They also admitted that there was "an issue" sending out notifications to me, and confirmed that none were sent.

So I guess just a warning to all - your database might be nuked at any time. I learned my lesson about not doing an offsite backup regularly. I guess the bigger lesson though is that Heroku should really be a last resort option for projects these days. RIP.

[ericpauley]

This whole debacle has been such short-term thinking from Salesforce. That after carrying these free projects for years they couldn't stomach more than 30d of data retention is just the icing on the cake.

In my view this has caused yet further reputational harm for Heroku, and is going to have a long-term effect on the bottom line from paid projects. The value prop of Heroku has always been being able to sleep at night, but clearly that's gone now.

[throwawaykai]

Herokai here. Unfortunately we had no choice on the data retention front — once we’ve disconnected your database, we aren’t ALLOWED to hold your data for more than 30 days. That’s part of the data scrubbing protocol that we agree to when you sign up. We fought hard for 90+ days internally, but in the end couldn’t get over the issue that we’d be in violation of our contracts with customers.

[xp84]

I assumed the value prop of being on Heroku in 2023 was purely "not having to do the pain in the butt to migrate your legacy app that you made on Heroku in the pre-2010 era" - in other words, I would be really shocked if Heroku got significant new business or any growth at all, now that it's just expensive AWS with some basic CI integration points.

I also assume Salesforce only bought them as a cash generator and has no interest in investing in it. So if they saved bottom line from this move, that's a win for them.

(Feel free to correct my assumptions if I'm very wrong)

[alexpetralia]

Salesforce is like the Midas touch of destruction.

[teaearlgraycold]

render.com

[codegeek]

"I learned my lesson about not doing an offsite backup regularly"

Heroku is a shitshow after the Salesforce takeover and not to shit on you because I know it really sucks. BUT please everyone, do offsite backups and test them. Please people. Please. If you have anything that is important, BACK THEM UP on your own outside of the provider.

Heck, we wrote our own script to backup RDS databases offsite as well even though RDS has backups and restore options. I want that database file.

[justin_oaks]

I spoke out loudly at a previous employer how it's really dumb to have our database and it's backups under a single AWS account. A single AWS account compromise, account issue (e.g. AWS shuts down account), or a disgruntled employee could result in the business being destroyed.

They took some half-hearted efforts to back up the data, but it was far from ideal.

Back up your data, and do it in different places so a failure in one won't affect any other copy of the data.

[Zababa]

I wish this kind of warning would come with a link to a website like dobackups.org that would document how to actually do them and test them. Something like a front page with different "profiles" like "Windows user", "Web application", "Database". Add to that some more precise docs, links to different solutions, maybe a bit of transparent sponsorship from backup companies, good practices like "how often to do backups", more specialized info for different use cases like family photos, legal documents (with user-contributed infos by countries).

I would like to do backups but it's already not really easy, even harder when you add the whole "figure out how to do backups" to it.

[Max-q]

After the Salesforce takeover? The one in 2010? So Heroku was good in like a year!?

[kaushikc]

Don't mind my curiosity but I wonder if you do offsite backup of your emails or just take it for granted that Gmail is a reliable enough service ? It is very important to me but I never even considered backing up my Google takeout data. The point I want to make is that service reliability is a critical factor when considering what to backup. It was expected from Heroku to do the right thing and be conservative in their approach.

[lamontcg]

Backups are apparently "too 20th century" for today's cloud-focused devops, you can just trust your entire business to someone else's procedures and if it all disappears in a puff of smoke, so do you.

(or just maybe we shouldn't have thrown out all the sysadmins with the bathwater)

[collectedparts]

This happened to me as well: projects with paid dynos but free databases, databases got nuked.

How Heroku missed this is beyond me. They managed to screw over paying customers in their broad attempt to stop freeloaders.

These are good accounts with credit cards on file. Why not just autocovert me to the lowest tier paid database?

FWIW I was able to get them to restore my databases. But I also had free Heroku Redis on one my projects and that, they assured me, is gone forever.

[addandsubtract]

I got "lucky" because I randomly checked the status page on my Heroku dashboard. It laid out which services were going to cost me after the switch date. I upgraded to the Eco tier and noticed that the DBs were still marked with a warning. So yeah, I would've been burned as well, had I not done that.

[fireworks]

Right? Like just charge me and tell me. Also, while I still think it is way too aggressive how they deprecated free in general, at the very least... maybe uh... I don't know, let me know if you are going to delete my database? Absolutely insane.

[sieabahlpark]

Redis is a memory store, you're the one who fucked up treating a cache as a persistence layer.

[marvinkennis]

Heroku sent me repeated messages that they would shut down my account due to inactivity. I was fine with that because finding and turning off the supposedly active Dyno was impossible. Guess what? Still charging me $27 a month for a server I can't even manage.

[dns_snek]

If this is true I think it deserves spotlight in a separate post.

[hot_gril]

A bit of a tangent, but my only real problem with Heroku involved a premium DB. Turns out that upgrading to premium enables high availability (HA) by default, and I don't even remember if you can disable it. HA replicates asynchronously to the standby master, so a master failover can cause a small amount of data loss. For my application, this was unacceptable, and I would have preferred unavailability instead (see CAP theorem). Today I have enough experience to check the fine print for that kind of detail, but anyway such a big change should come with big bold letters IMO.

[Edit: To this day I'm still puzzled by what I'm about to describe, so idk if it's Heroku's fault or mine.] I got a call from my colleague one day saying our database had gone back in time. Evidently we lost an hour of records. The code wasn't even capable of deleting rows, and nobody had direct DB access but me, so after leafing through the docs I suspected a failover event caused it. Premium DBs also let you roll back the DB to a previous point in time, and we were able to recover most of our data this way, like Back to the Future. If this really was a failover event, it's super weird if that the backup was more up to date than the standby master, and that a whole hour (rather than minute) was lost.

[nightpool]

Having a HA follower is the only different between Premium and Standard tiers, so I'm not really sure what else you expected them to do in this case. Like, premium-6 is 2x the cost of the standard-6 plan explicitly because of the HA follower.

[Andys]

Yes, AWS is similar with their DB offerings. You can discourage it from doing any updates/reboots (which causes a failover), but ultimately if they want to failover, they can at any time.

[londons_explore]

I wonder what architecture they use that can lose an hour of data?

Most architectures I see might lose a few milliseconds of writes in the typical case, and perhaps a second of writes in the worst case (which occurs when the master gets islanded with a couple of clients).

[93po]

Heroku banned my account with several income generating projects on it with zero notice. When I called them they treated me like a criminal and like they were doing me a massive fucking favor by even talking to me. I suspect I was supposed to get email notifications leading up to the ban to take corrective action but 100% did not get anything.

For context, someone put in malicious DMCA complaints against my website. Heroku did a shit fucking job (i.e. nothing) verifying the complaints. They provided no ability to dispute that it was bullshit. I took my business elsewhere, not that they care.

[celestialcheese]

Yup - this happened to me too. It was wildly frustrating, but they did send few emails looking back - it was just a really small project and it wasn't on my radar.

What pissed me off most is that I WAS paying for the account. I was paying $17/mo for redis and dynos, so there was an active card on file.

Why not just start charging for the postgres db, and only delete if there's no active billing?

Heroku was already a no-go for me with new projects, I just keep old things running in there since it's too much work to migrate off. This just cements that for me.

[mst]

> Why not just start charging for the postgres db, and only delete if there's no active billing?

Probably because the original terms you agreed to were written not anticipating Salesforce perpetrating this and so while I would not at all be surprised if the vast majority of customers in your position would've been entirely happy with it they probably didn't have a legal path to do so.

[tevon]

The reputation hit to Heroku is real. Its so too bad that they've made these changes, and stopped pushing new features.

I built my prior business on heroku and it was wonderful. Now with this current company I've switched to Render (through am looking for something else). For me the writing was on the wall for Heroku and it wasn't worth it to me to dig in there if new features and support wasn't a given.

Curious for anyone with an inside perspective. What happened? Heroku was so far ahead of the pack for awhile, then suddenly stopped staying a step ahead. Was this intentional? Why?

[strix_varius]

I explained a little here: https://news.ycombinator.com/item?id=34604723

[favorited]

Same exact thing happened to me. Out of nowhere, my (paid) app started failing. It was only after signing in did I see that the entire postgres database had been removed. No warning that this was going to happen, no ability to recover a backup. Just... deleted.

Not the end of the world for me, because 95% of the data I had in there had already been processed, but I did lose some. Complete joke of a service.

[yellowapple]

Something similar happened to me on IBM Cloud, back when it was called Bluemix. This was early in my career for an early-stage startup (that eventually shuttered - not due to this issue, but rather due to a lack of investor interest and us running out of runway as a result). Easy enough setup: staging and production, each with two Docker containers, one for the (Elixir/Sugar) app and one with the (Postgres) database.

Well, one day the production Postgres container just... vanishes. All the storage is gone. After weeks back and forth with IBM's support, they confirm that the loss is permanent. No explanation, no refunds, just "lol fuck you". Naturally, we didn't have backups yet, so the few users we did have now had to start from scratch.

I, too, learned my lesson about not doing an offsite backup regularly - and the bigger lesson that Bluemix should've been a last resort option. Funny enough, we had migrated to that from Heroku; we ended up migrating again to AWS (specifically: Elastic Beanstalk).

[marcelaguiar]

Same here. I have 2 big gripes:

- I did not receive any email warnings that this would happen. I have a valid email registered with them because I do receive non-promotional emails. However nothing for this. After getting in contact with customer service their rebuttal was that I also would've known by checking the forums/blog. Who on earth is doing that? It's a terrible response.

- They detached the DBs at the start of December. I wasn't going to fix my broken personal projects over the holidays. I told myself I would take a look come January. Which by that point it was past the 30 day grace period.

In the end, I've had to completely stop paying for the dynos. So Heroku lost business overall from me.

[pgrenn]

Same thing happened to me. My personal rails website, which I kept personal blog postings in a postgres database got completely lost. Most of those post date back to 2014.

I made some mistakes like not backing up my db, not keeping up on maintenance. But didn't think it would result in a complete loss of database considering I was still paying heroku. This app had been running since 2014. After this incident I removed my website from heroku.

[simonw]

In case this is useful, I recovered some lost content from my blog using the Internet Archive a few years ago and published notes about how I did it here: https://simonwillison.net/2017/Oct/8/missing-content/

[MzHN]

We also had some of our free databases on our Team tier projects nuked, which Heroku said would instead be automatically upgraded to lowest paid tier. No warning emails either, to anyone on our team.

My personal account did get a bunch of warning emails but nothing on our business accounts.

[cameronfraser]

Same happened to me, I didn't receive any email notification or anything, found out when things started failing.

[hyperman1]

I am wondering more and more if any trust in a single cloud provider isn't simply an unacceptable risk. The power balance means that a minor error on their side is generally business-ending for you. This is different from almost any other supplier, which can generally be replaced, even if the business is on hold for a few days. So any data recovery plan that should be able answer: what if the relation with our cloud provider disappears unexpectedly.

This is not a warning against only heroku. Google is famous for terminating without any recourse random accounts because they felt like it that day. Amazon and especially Microsoft seem more dependable, but even they had their share of business-killing behaviour.

So e.g. a backup with another cloud vendor is a requirement for almost any business. And of course, validate it. Easyer said than done at scale, of course, but even a partially failed backup is better than nothing.

[jfoster]

Even the seemingly independent ones can get acquired by Google and eventually killed.

Anyone remember Parse? It was quite successful until Facebook got hold of it and seemingly out of nowhere decided to destroy it.

[pattrn]

Same exact thing happened to a friend of mine who was just about to launch. He didn't realize that he had only paid for the dynos, but was using a free tier DB. No communication at all, and they wiped out his data. Needless to say, he is no longer a customer.

He's moved on to Render since then (and he's now backing up his data offsite). Painful lesson to learn, but at least he hadn't launched his product yet.

[rurp]

I remember being quite surprised when I first learned that Heroku was owned by Salesforce, because I had such a different impression of the two companies. I haven't used it in a few years, but Heroku used to be a great platform for certain types of projects. Unfortunately, the number of concerning stories I have heard about them in recent years has discouraged me from ever using Heroku for a future project.

[mtmail]

For reference here's the email another user received in November.

"Remind HN: Heroku will delete all free dbs and shut down all free dynos Monday" https://news.ycombinator.com/item?id=33755651

[lalopalota]

For me, the messaging in the announcements, dashboard, emails, and discussions here was clear and obvious that free databases were going to be deleted. If someone managed to miss all that, that's on them.

[dboreham]

They were charging money for thing A, that connected to (free) thing B, and rather than just begin charging the same customer, using same credit card, for thing B, they deleted it. I would love to see the recording of that zoom meeting...

[jfoster]

Is there any way for it to make business sense, or was it pure incompetence?

I wonder how many databases they deleted that they could have instead started charging for. Seems to be incredibly destructive for all involved. (SF shareholders, Heroku customers, etc.)

[numpad0]

Heroku also had weird business with Skeb.jp on 12/23-24 JST that was literally solved under the table - after Skeb redeployed to AWS. It seemed like a policy decision being made at Salesforce which owns Heroku. Whether such behavior adds confidence to their business users, I don’t know.

[sneak]

Anything stored in a cloud provider you're not paying a minimum of $10k a month should be assumed to be subject to evaporation without recourse at any time.

Make offsite backups.

For this use case, a raspberry pi 4 with a 1TB SD card hidden somewhere in your home with a cronjob is probably more than enough.

[benediktdeicke]

Oh crap! What a horror story.

We recently migrated from Heroku Postgres to Crunchy Bridge and can totally recommend it. So maybe set up your new database there instead.

[icey]

The same thing happened to me, but I was fortunate enough to catch it in time for them to restore it. Very disappointing because I was paying Heroku over $100/month for various things and didn’t realize an actively used project was on a free tier.

[jacobsenscott]

On heroku be sure you enable database backups with heroku pg:backups:schedule, and also script syncing those backups on a regular bases to some "off site" (non heroku) storage like your own s3 buckets. It is easy to get the URLs to you backups with `heroku pg:backups:url`.

I have no faith that if a rouge employee clicks the "delete app" button (because did you know to give an employee the ability to update the ssl cert on your web app you also need to give them permission to delete the whole damn app?) you'll ever be able to get your database back (although you might have a 30 day window to do so, but I wouldn't trust it.)

[latortuga]

This happened to me but I filed a ticket and they were able to restore it. I guess maybe it was within the 30 day window?

[carmenhchung]

This exact same thing happened to me with a client project I was working on - my client was paying $50/month for dynos, but we received no notifications about needing to pay for the database to avoid it being nuked!

On the upside, when I reached out to Heroku, they did offer to recover the database for me - but by this point in time, I'd already moved on with setting up a new one from scratch (with some changes from the original), so this wasn't particularly helpful.

[tchock23]

I also didn’t receive a notification about it, but some good samaritan HN user posted days before to give people a heads up. That’s the only way I found out.

[gloosx]

>your database might be nuked at any time

What is not directly possessed by you is not fully YOURS, and probably it is (un)clearly stated in the "free tier" terms of use.

Remember kids, when you encounter free cloud thing on top of the free host-able thing, just spend another day and host it yourself, cause you're just facing a sales funnel which will eventually collapse

[kuon]

Heroku was so nice "back in the day", it was expensive but served me well. I started having some issues before the acquisition so I moved away. I guess I was lucky for that.

They must have had some billing problem because I got charged for months after stopping everything, I was able to block the charges at the credit card level but they kept trying and I never had any answer nor explanation from them.

[ehPReth]

What a sad, money grubbing set of actions from Heroku. Tarnishing their brand forever to save a few dollars. Shame on them for doing this. There were so many better ways to deal with this and still save money in the longer run such as, https://news.ycombinator.com/item?id=33759178.

[nineteen999]

> I guess the bigger lesson though is that Heroku should really be a last resort option for projects these days.

Maybe people shouldn't, I don't know because I've never used it, but ...

> I learned my lesson about not doing an offsite backup regularly.

This is the bigger lesson, no matter whether you are using third party database hosting, or hosting yourself, whether it's a NoSQL database or a SQL one.

Once the data is gone, its gone.

[tflinton]

Heroku could have done better. At a bare minimum offering an option to download the existing data for 1-year or convert it to a paid db would have been better options and not brought this type of despise from the community.

I still use Heroku on a daily basis and wouldn't say this has caused me to re-evaluate my decision to stay with them, but then again i'm not using anything free from them.

[kakarukeys]

Not surprised to hear that. This happened to me in 2019, and for the whole Heroku account not just 1 database. See https://news.ycombinator.com/item?id=21969358.

They were going downhill since Salesforce takeover.

[frietzkriesler2]

Bought some HDDs with some left over end of quarter money, found an old dell server, slapped em in installed it in a switch closet, and have the cloud perform backups to this local closet server. I even do this at home. I don't trust these guys with anything. Sorry for your loss

[deepsun]

I remember in Google one project sent some announcement to incorrect recipients. Upon investigation it turned on that the bug was in all SQL queries that selected relevant customers, but it was broken for years and no one complained before our investigation.

[rovr138]

Well, I moved everything I had to a VPS for the time being because I left it to the end.

So I take this opportunity to ask, what alternatives have people moved to? I really haven't gone back to look what's out there.

Not looking for free, just alternatives to review.

[dcow]

You can host static sites on DigitalOcean's app platform for free. Their lowest tier offering for non-static sites is like $5/mo. Might be worth checking out.

[xp84]

For what kind of an app? If I was tasked to put an app someplace today, the answer would depend if it was easily containerized, and what kind of scale you're talking about.

For small apps, if containerized, I would host on Google Cloud Platform using App Engine Flexible Environment: https://cloud.google.com/appengine/docs/flexible Add on their managed database, of course.

But also, random VPSs are so cheap that if it works fine as is, you could do a lot worse than just running it on Linode, Digital Ocean, Hetzner, etc.

[donnieashok]

I went full blast into SQLite, agreed there's a migration job in between. But it's totally worth it. In fact my app is now 5x faster.

[ioseph]

Happened to myself as well, luckily it was just our staging environment but it was while I was on leave. I feel one should reasonably expect their configuration to be stable to 3 months and 30 day data retention is just insulting.

[bryanrasmussen]

reading this I keep thinking the real questions are:

1. probably the notice sending and db deletion is two separate teams or responsibilities. Are they?

2. Did people know there was a bug in notice sending so some notices were not being sent, all notices not being sent? I ask this because generally in places I've worked where notice sending was an important part of things you knew if there was a bug and notices were not being sent. But maybe it wasn't that important for Heroku. Maybe it was not known that notices were not being sent for a while - or was it known immediately but things on other parts of business chugged along anyway.

3. If they knew notices were not being sent and they went ahead and deleted db anyway, seems messed up, but that would probably be ok with people if they had data retention for people who did not get notices sent.

4. The whole thing your stuff can be deleted at any time without telling you is basically probably true almost everywhere in that notice sending can have a bug and deletion of stuff is probably not adequately tied to notice sending so that if notice not sent automatic deletion is stopped. Which I'm thinking is probably everywhere - if you work somewhere with automatic deletion and a notice sending module - what happens? Is this scenario handled?

5. answer to this is probably not, but is there a legal issue if notices not sent and stuff deleted, issue might be if some notices were sent - if account A gets notice about deletion and is thus able to act on it and account B does not get notice and is thus not able to act on it there might be a ground for action. Probably not, but when something seems unfair there might be a law that can be stretched to fit it.

[mst]

My guess would be that a tiny percentage of accounts didn't get their notices sent, nobody complained about it because they didn't realise they were supposed to be getting them (or at least for whatever reason no such complaints were escalated to the people who would go "oh shit" upon seeing one) and some unfortunate combination of technical factors meant whatever internal monitoring existed didn't pick up the omissions.

Note that this is not to say the end result wasn't an indefensible disaster, only that disasters seldom have only a single cause and the above is my best uneducated guess at how things came together to cause this one.

[reiderrider]

Similar issue with billing post the Salesforce acquisition. We’re on a Enterprise Heroku plan but the billing could only be handled within a Salesforce account (which we don’t have). Autopay disconnected and our app went offline for 3 hours until we could confirm payment over the phone. Billing went from self serve to having to call in to provide a credit card over the phone.

[brightball]

Best option is https://www.crunchydata.com/ IMO.

[coffeeblack]

Never trust any company or hardware. Either can fail you at any moment. Always keep your own backup. Still sucks though.

[thinking001001]

This is probably the best advice

[swader999]

I'm curious how this affected your projects on paid dynos, can you provide more details around that?

[fireworks]

So I did not realize this at first (would have helped if I were, you know, emailed some information about this), but my project was still using a free Postgres addon despite being on a paid dyno setup. So that I suppose was nuked along with the general free tier deprecation.

[m00x]

I don't really understand. You were paying for the dyno, but they deleted your instance anyway?

[danpalmer]

They deleted the database.

On Heroku, databases and dynos are completely distinct entities, each with their own payment plans.

When you set up a dyno and “add” a database, as most people do, it’s really easy to think that they are part of the same plan, and conclude that the “pro” or paid dyno is actually the combo of the dyno and database. They are not.

I understand why they did this, databases can be shared between applications, that’s handy, but this is a sharp edge.

[riebschlager]

They were likely paying for a web dyno, but were using a free database instance.

[jack_riminton]

Free db but paid dyno in this case I think

[kragen]

does seem like a cautionary story that should rule heroku out for anything except quick experiments

[yonrg]

I can just second most of those comments here. One thing should be clear: It doesn't matter if you have a payed or free plan with someone who stores your data. Always make backups! There are so many things what can go wrong

[nzoschke]

Heroku also has a free or cheap tool to back up databases to s3 and download or copy the backup.

Totally understand expectations that a managed database service will keep your data but if it’s truly mission critical take backups…

[menzoic]

Times like this are great opportunities for growth. This can help you build consistent habits in the future to have more durable storage practices.

[dtjohnnymonkey]

Mine was also deleted recently, I just had assumed I wasn’t reading their emails carefully, but good to know I wasn’t the only one.

[userbinator]

"Cloud is just a fancy word for someone else's computer."

[sebastien_b]

All your databases are belong to /dev/null

[robbiep]

I also had this happen - unbelievable really

[anonym29]

Not your hardware, not your database.

[spuz]

He said he used a paid dyno

[JEDI-HACKER]

Check your junk mail folder.

[javawizard]

> They also admitted that there was "an issue" sending out notifications to me, and confirmed that none were sent.

Junk mail wasn't the issue in this case.

[purpleblue]

You didn't have a full backup of your database? That's on you. Disaster Recovery is an important part of any project. Heroku could have blipped out of existence by accident and you would still have the same problem.

[gwbas1c]

What frustrated me about Heroku nuking free projects was that the paid version was, IMO, too expensive. (Granted, I understand that the free tier was probably a money sink for them.)

$5-10 a month is pocket change in some contexts, but for a hobby or a one-off project, it's a huge chunk of change.

I really wish they offered a lower-cost tier.

[Towaway69]

My personal take is that I'd rather work with heroku than dealing with aws or gcs.

Sure there are others out there that provide the same easy of use, I am just too lazy to switch ;)

OTOH I've never had them wipe a database on me, so I guess that would be the motivation to move on.

[shortcake27]

Or let me share dynos across all my projects. $5-$10 per month for all projects, sounds good! $5 per month for each project adds up very quickly.

[m00x]

That's 2 cups of coffee, or 1 latte. It's objectively not a huge chunk of change unless you live in a very LCOL area.

You can also go to any competitors, which have their smallest tier around the same price. Linode has a $5/m tier.

You could also buy a Raspberry Pi for $50 and run it in your closet.