Software System Administration

Gmvault: gmail backup (Ubuntu install and setup)

(edit 8/20/2012: Originally written for v1.5, this has been tested and works fine as of Gmvault v1.7-beta, so just change things accordingly and install away! I’ve updated the links below.)

I’ve been using getmail for over a year to backup my Gmail account. As far as I know, it’s been working, but it just grabs email over POP, dumps them into an mbox file, and that’s that. It’s a fairly basic procedure, but since it just blindly downloads emails, getmail doesn’t account for ones I’ve deleted, and I don’t know if it keeps any label information.

So along comes Gmvault, the new kid on the block with fancy restore features and multiple OS capability. Let’s give it a shot.

Python should be installed on a default Ubuntu machine, but let’s install a few other pre-requisites as well. I’m using Ubuntu 12.04 LTS (Precise Pangolin).

sudo apt-get install python python-setuptools
sudo easy_install -U distribute

Then download the source package for Gmvault:


Unpack it:

tar xvzf gmvault-1.7-beta-src.tar.gz

Then install!

cd gmvault-1.7-beta/
sudo python install

The basic setup is about as simple as it comes:

gmvault sync

The setup process will take you through an authentication process with your email account.

Since I ran the install on an Ubuntu headless server, it loaded the text-based w3m browser, which I was able to login to my account, but then was unable to approve Gmvault to access my email. Pressing ‘q’ quits the browser and you’ll be presented with with an OAuth link you can copy into a normal browser and authorize Gmvault that way. (Note: you can manage what apps can access your Google Account at any time)

Once Gmvault is given the okay, it’ll start churning away downloading emails. Give it a few minutes (or hours) to finish, and feel better knowing that your hoard of oh-so-important emails are now tucked away in yet another location.

There’s a few other parameters you can pass to the program, which will allow you to change the vault database location, provide a user/pass rather than the OAuth token, and to run custom synchronization of certain emails. Take a look at the in-depth documentation on Gmvault.

After running the initial sync, you can run the following command to run a quick sync (only the last 2 months are scanned):

gmvault sync --type quick

Restoration of emails appears to be straightforward (although I haven’t tried it) and emails are stored in month folders as individual .eml.gz and .meta files. The gzipping appears to be fairly efficient, as the Gmvault database reports to be about 2/3rds the reported size of my Gmail inbox. Not bad at all!

Also–for the sake of completeness, a few days after I wrote this I found out about BaGoMa, which is another Python-based utility to backup your Gmail. Seems to follow a similar path as Gmvault, but I like Gmvault’s ability to use OAuth rather than have to provide my password each time.

So far, Gmvault appears to have a simple and easy to use product that fits my needs. Good work!

System Administration

Forwarding Yahoo! Mail to Gmail (For Free!)

I’ve got way too many email addresses. Free email accounts opened up as a teenager that I don’t want to get rid of because they hold some strange nostalgic value. Hotmail, Gmail, Yahoo…I’ve even got an account with RocketMail, which eventually became Yahoo! Mail.

So consolidation is my other option.  I’m using Google Apps for Domains which essentially gives me Gmail on my own domain.  Most of the free email services have some mechanism which I can use to forward to my Gmail account, but Yahoo! requires their paid service.  No fun.

If I was using a desktop email client I could just implement YPOPs! or FreePOPs. But I’m not.

Linux to the rescue! Using Exim and FetchYahoo I can now periodically check my Yahoo! Mail accounts and redirect them to my Gmail account. FetchYahoo is great because it provides the mechanism to forward emails to another email address; the products mentioned about don’t appear to have this feature.

Here’s how I did it, using Ubuntu 8.10 (Intrepid Ibex).

Reconfigure Exim4 to use Gmail SMTP (Note: POP must be enabled in your Gmail settings). I used the instructions I found on TechSutra, which I’m not going to reproduce here because I’m lazy.

Remember to comment out or remove any other smarthost defined with “domains = ! +local_domains” as well as any other authenticators where “public_name = LOGIN”. These were both defined for me and I originally missed these steps.

Ensure that Exim4 is working. See step 3 from TechSutra’s instructions.

Download and install FetchYahoo (2.13.3 as of this writing). Alternatively, install from the repos (it’s an older version though):

sudo apt-get install fetchyahoo

Test FetchYahoo. Replace <username> and <password> with your, well, username and password.

fetchyahoo --mailhost=localhost --username=<username> \
--password=<password> --onlylistmessages

You should see a list of the emails in your inbox. If it bombs out, you might try logging into your Yahoo! Mail and changing the interface from Classic to New and vice-versa.

Try forwarding to Gmail using FetchYahoo. You can limit the number of emails it fetches by using –maxmessages=N where N is the maximum number of messages to grab, but I only had a handful of emails in my inbox at the time.  Once the emails are processed, they’ll end up in the Trash, so if there are problems, you can just move them back to the inbox and try again.

fetchyahoo --sendto=<username> --mailhost=localhost \
--username=<username> --password=<password>

You should see something like the following:

Only forwarding e-mail, local delivery turned off.
Logging in securely via SSL as <username> on Wed Apr 15 00:04:15 2009
Country code : us    FetchYahoo! Version: 2.13.3
Successfully logged in as <username>
Marking messages read on the server

Fetching mail from folder: Inbox
Getting Message ID(s) for message(s) 1 - 5.
Got 5 Message IDs
Finished downloading 5 messages.
5 message(s) have been deleted.

You can monitor the Exim4 logfile to see if the emails are being sent properly:

tail -f /var/log/exim/mainlog

Check your Gmail account for the forwarded emails. Now the only glitch I’m having here is that forwarded emails seem to be coming into All Mail rather than Inbox and are already marked as read.  If anyone has any ideas on this, please let me know.

If all is well at this point, then congratulations! If all is not well, then I don’t know what to tell you, slick. It worked for me.

Create FetchYahoo configuration file. FetchYahoo allows for an /etc/fetchyahoorc file or ~/.fetchyahoorc file rather than passing all the options via the command line. From the FetchYahoo source folder, copy fetchyahoorc to your home directory:

cp fetchyahoorc ~/.fetchyahoorc

Then edit the file, substituting your own values as applicable.  The most important values are:

username =
password = yahoo-password
use-forward = 1
mail-host = localhost
send-to =

If you’d rather store a hashed password than the plaintext, you can run

fetchyahoo --md5hex

which will give you a md5 hash of your password suitable for storing in the configuration file.

Save and test the config file by running fetchyahoo with no parameters.

Schedule to run periodically via cron or use the daemon mode. I’m using cron because I’ll probably only check these accounts every few days and I don’t need the process just hanging around in the meantime.  The –repeat-interval=N command line parameter (or just repeat-interval = N in the config file) is all you need to run the process as a daemon.

And that’s it!  So until the Yahoo! folks decide to come around and allow for POP3 access or some other mechanism to get email out from their stronghold, this is what I’ll use to get my coveted bits of spam to Gmail.

Update: The original procedure described above ended up dropping forwarded emails from Yahoo into the All Mail folder rather than the Inbox due to the fact that I was sending to and from the same Gmail account.  Since I implemented this procedure using Google Apps for Domains, I simply created a generic account from which I am sending emails, and all is now well with the world.