Gmvault: Backup and restore your Gmail account

[joshstrange]

I have been using this for years and it's great software. One tip, store each different email account in it's own "database" my crontab looks like this:

    #Email backup
    0 0,6,12,18 * * * /usr/local/bin/gmvault sync -d /XXXXXXX/email-backup/joshstrange --resume [email protected] >> ~/mail-backup-logs/joshstrange.com.log 2>&1
    0 1,7,13,19 * * * /usr/local/bin/gmvault sync -d /XXXXXXX/email-backup/otheremail --resume [email protected] >> ~/mail-backup-logs/otheremail.com.log 2>&1
    ....

I sync it all locally to my house then back it up to Dropbox as well. The reason to store them in different datebases is you cannot "filter" them out when restoring so if they all go to the same DB if you restore you are restoring ALL your email across all accounts to one new account.

[nicholasjarnold]

Chiming in to second this comment for anyone who is skeptical of using gmvault. I too have used it for years with great success. Thanks to the author for creating it!

[laumars]

Thank you for your advice, that's handy to know.

What's the database format it uses, sqlite3? (I tried looking on the repo but couldn't any obvious reference)

[handruin]

I'm interested to know what the pros and cons are of this utility vs using the Google takeout functionality? I like the idea of this project but I don't know what it would gain me over Google's native export? Is it the restoration that's missing from Google's service?

https://takeout.google.com/settings/takeout

[xur17]

My coworker and I both tried this yesterday, and today we both received an email with this error message: "Sorry, we encountered a problem when creating your Google data archive."

[amelius]

Probably the most useful aspects are that you can run it in batch mode (without supervision), and that you can backup incrementally.

(Note that these aspects may also make the tool a little more dangerous to use.)

[e40]

I'd like to know, too. On the surface, it would seem that scripting is the main benefit to it, but there might well be other pros/cons.

[tonyplee]

I just started a takeout session on all my google drive, emails data too.

I wonder how to use them once I have the backup data?

[emilburzo]

After yesterday's post[1], I've setup a cron job to regularly backup Gmail/Google Drive.

I'm using Gmvault for Gmail (emails and chats), pretty effortless and with enough features for me, especially the export to maildir.

And rclone[2] for Google Drive, in "create a local mirror" mode (sync).

Any other tools to backup your Google life?

[1] - https://news.ycombinator.com/item?id=12972554 [2] - http://rclone.org/

[jdc0589]

I am actively trying to get off of dropbox. I really want a similar native application experience that syncs to-and-from S3. Not a cron job using the s3 cli, not something that only works on osx/windows/etc... So far owncloud enterprise is the only polished looking solution I've found, but thats a bit overkill...

[wfunction]

WARNING/fun fact: it doesn't download all emails properly. Last time I tried it, it seemed that when the Gmail server randomly closed a connection (or maybe some other time, but I think it was in these instances), the program would just keep whatever partial results it had and then move on to the next email. Which meant I had a lot of partial emails on my drive (only a small fraction of all the emails, but still), and no way to detect them.

[ljosa]

I have been using both GYB (https://github.com/jay0lee/got-your-back) and mbsync (http://isync.sourceforge.net/mbsync.html) for this. How is gmvault better/different?

[wyldfire]

Why do you need to use both? Or are you saying that they're both equivalent and you'd had success with each?

[wielebny]

I've been using gmvault for only one feature: ability to fetch e-mails that match a certain filter.

[tyingq]

Reading over the details, gmvault supports two things I don't see in GYB and one thing mbsync doesn't support:

- XOauth

- Sync to another gmail account (incrementally)

Edit: Also supports only backing up emails with a specific gmail tag.

[pawadu]

Given how easy and carelessly Google can close your account and ruin your digital life, I guess periodic backups of your cloud accounts will soon be considered a good practice.

[scholia]

> I guess periodic backups of your cloud accounts will soon be considered a good practice.

It has been good practice for a couple of decades. Of course, getting people to follow good practices is another matter....

[adam12]

I'm afraid this tool might trigger my account to be disabled.

[Demcox]

This - and having no unique data anywhere. Having your files distributed this day and age should be general approach.

[bit_logic]

Data is only half the problem. The other is that email address has become the "primary key" for everything. Banking websites, random forums, everything. It's sometimes impossible to change it because it's the "primary key" for identity on that website. And email addresses are not portable like phone numbers.

[eigenvector]

Email addresses are portable to any webmail, mail server or other email infrastructure provider if you have your own domain and then forward to the service of your choice. This way you can use Gmail if you like or maintain your own full email stack, or anything in between, while still addressing your mail to something you control.

[laurent123456]

Since Gmail supports imap and pop3, one could simply use any proven email client to backup the emails. I don't think a special tool that or may not work is needed for this.

[andybak]

Gmail tags don't map cleanly on to IMAP folders. The abstraction is leaky especially for backup/restore.

[cookiecaper]

Which "proven email client" do you recommend? There's an Import/Export extension for Thunderbird, one of the last standing desktop mail apps, but it's not good at handling huge, multi-thousand message exports.

gmvault is purpose-built and quite simple. I've used it before with success. It backs up your whole mailbox in one command. Why should we fiddle with a desktop mail client?

[AdmiralAsshat]

The 'restore' feature looks nice. But would it be of any value if Google decides to close your account? That is, can I take my backup emails from Gmail and 'restore' it to [email protected] in order to migrate all those emails over?

[bshep]

From reading the docs it seems possible

Go to the website click on documentation then indepth then restore.

You can also export to other formats ex. mbox

[timdafweak]

Good question, I wonder if anyone’s tried this before. It would be invaluable if that were the case.

[kingnight]

My backup of Gmail is to use Mail.app via IMAP + download all attachments.

I then have a backup of my computer -> 1. Time Machine + 2. Arq->S3/Glacier

Given that this keeps mail locally in a constantly readible format (offline, copied in mbox)... is there something missing in my basic solution that this cli utility adds?

[jxy]

Or any IMAP client that saves messages locally.

I guess this utility benefits people who only use gmail's web interface.

[ajennings]

Here's one I wrote: https://github.com/abjennings/gmail-backup

Not as full featured (can't restore), but it's just a 77-line Python script. You could audit it yourself to make sure it doesn't upload your creds to another server.

[lexalizer]

Please don't forget to encrypt your backups in case your local drive gets stolen or your cloud backup service is hacked.

[dharma1]

doesn't http://www.google.com/takeout include gmail?

[cdubzzz]

Yep! And much more. But gmvault does also restore emails to an account, interestingly.

I am (slowly) working on a project to pull some statistics from Takeout's mbox file for Mail. Also want to play around with the Location History, Chrome data and Hangouts exports.

[tedmiston]

Yes but I don't believe there's any way to automate Takeout. Also each new takeout is a full backup vs incremental.

[lima]

No

[hengheng]

Since yesterday I have a few Google Takeout zip files in my backup ( https://takeout.google.com/settings/takeout ). I've used gmvault in the past, but this looks superior from the outside. Haven't delved into the data I'll admit.

[cdubzzz]

I am working on digging around this data right now, actually. Some details on my website (see profile) and I will post on HN about it at some point when I get things further along (also want to look at Location History and Chrome data exports).

[amq]

Why should I use this over an email client like Thunderbird?

[Arkanosis]

I find it incredibly easier to configure and run periodically in a cron job. Plus I've never had even the slightest stability issue with gmvault whereas Thunderbird (which I use anyway to read emails at work) has bad days from time to time.

Additionally, the output format (gzipped plain email + metadata) looks very convenient for indexing / analysis; something I'm dreaming of for a long time.

[ozim]

I think there is no point, but if someone uses only web interface this might be valuable.

[tajen]

Do you use IMAP? Do emails get fully downloaded in IMAP?

[hatchoo]

I have used this in the past to backup email accounts of resigned employees so we continue to stay well within the maximum number of active accounts for our free Google Apps.

It generally works fine and it allows you to restore the emails to a different account name. (I sometimes temporarily restore an account to search for old emails). It seems to have some issues with restoring accounts with a lot of large emails (large or multiple attachments) especially those that have reached the 15GB quota.

[hobarrera]

> Handle all Gmail IMAP hiccups.

It's sad that after all these years, google still hasn't gotten around to properly implementing IMAP.

[fgpwd]

Is it considered safe to leave your oauth ID and Secret public in open source projects?

[ajonit]

I have been using Spanning backup for several years now. It backs up our Gmail, documents , calendars, contacts, sites It is one of those set-it-and-forget-it type services. The company is acquired by EMC now. The service costs us around $35/year/email account; for us it is small price for peace of mind. I wrote a review of them few years back - http://reviewofweb.com/gmail/backupify-vs-spanning/

[aq3cn]

I have been planning to switch to mutt, this tool will also come in handy.

Are there similar tools for MS Outlook or Protonmail or if it is possible to modify gmvault to work with either one?