Categories
Software System Administration

Gmvault: gmail backup (Ubuntu install and setup)

(edit 8/20/2012: Originally written for v1.5, this has been tested and works fine as of Gmvault v1.7-beta, so just change things accordingly and install away! I’ve updated the links below.)

I’ve been using getmail for over a year to backup my Gmail account. As far as I know, it’s been working, but it just grabs email over POP, dumps them into an mbox file, and that’s that. It’s a fairly basic procedure, but since it just blindly downloads emails, getmail doesn’t account for ones I’ve deleted, and I don’t know if it keeps any label information.

So along comes Gmvault, the new kid on the block with fancy restore features and multiple OS capability. Let’s give it a shot.

Python should be installed on a default Ubuntu machine, but let’s install a few other pre-requisites as well. I’m using Ubuntu 12.04 LTS (Precise Pangolin).

sudo apt-get install python python-setuptools
sudo easy_install -U distribute

Then download the source package for Gmvault:

wget https://github.com/downloads/gaubert/gmvault/gmvault-v1.7-beta-src.tar.gz

Unpack it:

tar xvzf gmvault-1.7-beta-src.tar.gz

Then install!

cd gmvault-1.7-beta/
sudo python setup.py install

The basic setup is about as simple as it comes:

gmvault sync your_email@gmail.com

The setup process will take you through an authentication process with your email account.

Since I ran the install on an Ubuntu headless server, it loaded the text-based w3m browser, which I was able to login to my account, but then was unable to approve Gmvault to access my email. Pressing ‘q’ quits the browser and you’ll be presented with with an OAuth link you can copy into a normal browser and authorize Gmvault that way. (Note: you can manage what apps can access your Google Account at any time)

Once Gmvault is given the okay, it’ll start churning away downloading emails. Give it a few minutes (or hours) to finish, and feel better knowing that your hoard of oh-so-important emails are now tucked away in yet another location.

There’s a few other parameters you can pass to the program, which will allow you to change the vault database location, provide a user/pass rather than the OAuth token, and to run custom synchronization of certain emails. Take a look at the in-depth documentation on Gmvault.

After running the initial sync, you can run the following command to run a quick sync (only the last 2 months are scanned):

gmvault sync --type quick your_email@gmail.com

Restoration of emails appears to be straightforward (although I haven’t tried it) and emails are stored in month folders as individual .eml.gz and .meta files. The gzipping appears to be fairly efficient, as the Gmvault database reports to be about 2/3rds the reported size of my Gmail inbox. Not bad at all!

Also–for the sake of completeness, a few days after I wrote this I found out about BaGoMa, which is another Python-based utility to backup your Gmail. Seems to follow a similar path as Gmvault, but I like Gmvault’s ability to use OAuth rather than have to provide my password each time.

So far, Gmvault appears to have a simple and easy to use product that fits my needs. Good work!

Categories
System Administration

Ubuntu 12.04 + rkhunter 1.3.8 = false positives!

rkhunter (Rootkit Hunter) is a wonderful tool to assist in securing your system, but it becomes much less helpful if it constantly reports false positives.

After updating your rkhunter.conf file appropriately, you may still be left with the following error:

Warning: Hidden file found: /dev/.initramfs: symbolic link to `/run/initramfs’

This file can’t be whitelisted because it’s a symlink, and rkhunter 1.3.8 doesn’t check for this condition. The 1.4.0 release fixes this bug, but it’s not available from the Ubuntu repos. So you’ve got a few options.

  • Install rkhunter 1.4.0 manually.
  • Download the rkhunter 1.4.0 tarball and extract the executable and conf file yourself
  • Patch the rkhunter executable

I opted to patch the rkhunter executable, which really amounted to adding a few lines (one actual test case line + a few comments). Follow along!

Open up /usr/bin/rkhunter and move down to line 834. You should see the following:

Now, after line 846, you’ll want to add the following lines:

So it looks like this:

Save the file and then open up /etc/rkhunter.conf and add the following line:

ALLOWHIDDENFILE="/dev/.initramfs"

Save, and finally, update the rkhunter properties:

sudo rkhunter –-propupd

There you go! There are a few other items that need to be whitelisted in the conf, but they should all work with the rkhunter 1.3.8, so it’s just a matter of updating the file accordingly.

Thanks to reinhold and Tim Miller Dyck for the patch implementation information and unSpawn for the actual patch. Here’s hoping to see an actual backport fix into the Ubuntu repos.

More information:

 

Categories
Software System Administration

Upgrading to NConf 1.3.0 on Ubuntu 11.10 Oneiric Ocelot

Congrats to the NConf crew for reaching the 1.3.0 milestone! Two years in the making, but I’m excited to see the project continuing to move forward.

It’s been over a year since I originally installed and configured Nagios and NConf 1.2.6 on my Lucid Lynx server. In the meantime, I’ve upgraded the hardware and moved to Ubuntu 11.10 Oneiric Ocelot, mostly to keep up with the current releases. Not a whole lot has changed for what I use this server for, but keeping up with the Nagios releases in the repos has been convenient.

If you’re following along with this guide but starting fresh with Nagios and Nconf, you’ll probably want to take a look at my previous guide and the NConf installation documentation to get some background information. I’ll try to provide some notes for those of you doing a fresh install, but since I’m just upgrading, I can’t make any promises.

Backups!

Log into your server and make them backups!

sudo cp -rp /var/www/nconf /var/www/nconf.bak
mysqldump -u root -p nconf > ~/nconf_sql.bak

I also backed up my Nagios configs

sudo cp -rp /etc/nagios3 /etc/nagios3.bak

Download and Unpack

Wget or otherwise download the latest version on Nconf. Wget has that issue with following SourceForge links so you may need to rename the file.

Unpack the files and then copy them to where you installed NConf before, or do it all in one step:

sudo tar xzvf nconf-1.3.0-0.tgz -C /var/www

Then fix the permissions for the NConf folder, setting them back to your apache user:

sudo chown -R www-data:www-data /var/www/nconf

Upgrade

Browse to http://<yourserver>/nconf/UPDATE.php

Follow the prompts to proceed with the interactive update. NConf’s documentation on the update process has screenshots that I won’t repost here.

After the upgrade, you’ll need to remove some files and folders. Browse to your NConf folder and nuke them. I also had to remove the call_ajax.php file, which wasn’t in the original documentation.

cd /var/www/nconf
sudo rm -rf INSTALL/ UPDATE/ INSTALL.php UPDATE.php call_ajax.php

There’s been a few changes/additions to the NConf conf file, so open up config/nconf.php and add the following lines:

define('CHECK_UPDATE', 1);
define("DEBUG_GENERATE", 3); # [1=ERROR|2=WARN|3=INFO|4=DEBUG|5=TRACE]
define('CHECK_STATIC_SYNTAX', 1);

Also, set the ALLOW_DEPLOYMENT variable to 1 to enable deploying configs directly from NConf and comment out the other CONF_DEPLOY_ variables, as they aren’t used anymore.

define('ALLOW_DEPLOYMENT', 1);
#define('CONF_DEPLOY_URL', "https://webserver.mydomain.com/incoming_config.php");
#define('CONF_DEPLOY_USER', "deployUser");
#define('CONF_DEPLOY_PWD', "deployPass");

At this point, you should be able to log into NConf 1.3.0 and look around.

Migrating Deployment

With NConf 1.2.6, I used the ADD-ONS/deploy_local.sh script and a cron job to deploy configuration changes to Nagios. In 1.3.0, they’ve integrated this a little better into the core of the application, but it required some adjustment in my setup.

Start off by coping the deployment.ini sample configuration file to the config folder:

sudo cp /var/www/nconf/config.orig/deployment.ini /var/www/nconf/config/deployment.ini

Open up the config/deployment.ini file, uncomment the LOCAL deployment section except for the [copy nagios.cfg] section and change the target_file variables to reference the nagios3 folder where Ubuntu places the config files.

Also, add the reload command (sudo /etc/init.d/nagios3 reload) at the bottom of the [copy global config] section. When you’re done the file will look similar to the following:

;; LOCAL deployment ;;

[extract config]
type = local
source_file = "/var/www/nconf/output/NagiosConfig.tgz"
target_file = "/tmp/"
action = extract

[copy collector config]
type = local
source_file = "/tmp/Default_collector/"
;target_file = "/etc/nagios/Default_collector/"
target_file = "/etc/nagios3/Default_collector/"
action = copy

[copy global config]
type = local
source_file = "/tmp/global/"
;target_file = "/etc/nagios/global/"
target_file = "/etc/nagios3/global/"
action = copy
; *** add the line below
reload_command = "sudo /etc/init.d/nagios3 reload"
; *** keep this section and the rest of the file commented out
;[copy nagios.cfg]
;type = local
;source_file = "/tmp/static_cfg/nagios.cfg"
;target_file = "/etc/nagios/nagios.cfg"
;action = copy
;reload_command = "sudo /etc/rc.d/init.d/nagios reload"

This process replaces the deploy_local.sh script, but we’ll need to allow the web user to issue the Nagios reload. To do that, we need to modify the sudoers file:

sudo visudo

Then add the following lines:

# Allow www-data (apache) user to reload nagios
www-data ALL=NOPASSWD: /etc/init.d/nagios3 reload

Save and quit.

Finally, remove the entry in the root crontab entry that launched the old deploy script:

sudo crontab -e

And remove the line reading:

* * * * * /var/www/nconf/ADD-ONS/deploy_local.sh

Deployment Testing and Cleanup

Let’s make sure the new deployment procedure works. Back in the NConf GUI, click Generate Nagios config. You should hopefully see something like:

[INFO]  Starting generate_config script
[INFO]  Generating global config files
[INFO]  Generating config for Nagios-collector 'Default Nagios'
[INFO]  Ended generate_config script

If the config files generate successfully, you’ll see a button aptly labeled ‘Deploy’.

Click Deploy, and with any luck you’ll be presented with three OK messages in NConf  and your Nagios Event Log should show a SIGHUP message.

Relax, and enjoy some JQuerified NConf 1.3.0 goodness! Gold star!

I did notice that the original deployment method left a bunch of old config tgz bundles, so if you’re feeling like cleaning those up, just remove the whole lot:

sudo rm /var/www/nconf/output/NagiosConfig.tgz.*

Update (6/25/12): You’ll want to remove the extended_host_info.cfg and extended_service_info.cfg files from your target config directory (/etc/nagios3/Default_collector/), or you might get weird problems trying to deploy later on. I just found this out six months later.

Troubleshooting Notes

  • The biggest problems I had were just remembering to reference nagios3 rather than nagios for paths and commands.  Make sure you reference the correct locations. This caused me all sorts of weird grief with the sudoers file, and I didn’t catch it until later.
  • While NConf will run without the additional variables in config/nconf.php, it’ll log warnings until you add them.
  • I had to remove an extra file (call_ajax.php) before NConf would run. The file has been replaced with call_file.php which apparently fixes a security issue. At the time of this writing, removing this file wasn’t in the docs, but it complains until you nuke it.

Fresh Install of Nagios and NConf

From what I can tell, you can follow all the steps in my old guide for installing 1.3.0 up until you get to Configuring NConf to Deploy Nagios Configurations Automatically. At that point, follow the Migrating Deployment instructions from this guide.

Please leave a comment and let me know if you do a fresh install. I’d be curious to know how it works for you.

More Information

 (edit 6/25/12 to add additional file cleanup instructions)

 

Categories
System Administration

Nagios and NConf on Ubuntu 10.04 Lucid Lynx

As of December 2011, NConf has released version 1.3.0 which changes the deployment methods for configs. Take a look at my NConf 1.3.0 upgrade guide!

Have you been wondering about how to get Nagios and NConf setup on your Ubuntu server?  Well, I took some notes the last time around, and here they are, for anyone that may find them useful.  Here we go!

(disclaimer: don’t screw up your system.)

Prerequisites

If you didn’t check the box to configure your box as a LAMP (Linux, Apache, MySQL, PHP) server when you first setup your box, do that first with:

sudo apt-get install apache2
sudo apt-get install mysql-server mysql-client
sudo apt-get install php5 libapache2-mod-php5

If you want phpMyAdmin (for web-based MySQL administration):

sudo apt-get install phpmyadmin

Installing Nagios

Install Nagios with

sudo apt-get install nagios3

It should install all the necessary dependencies. Follow the prompts to get it configured and test it out by browsing to http://<yourserver>/nagios3

If you would like to monitor Windows servers using NSClient++ (which I am doing), also install the NRPE plugin with

sudo apt-get install nagios-nrpe-plugin

I felt like I ought to backup the config files before I started messing with them, so I backed them up with

sudo cp -r /etc/nagios3 /etc/nagios3.backup

Installing NConf

NConf provides a web-based frontend for configuring Nagios.  There’s no package in the Ubuntu repositories for it, but you can download it pretty easily.  Version 1.2.6 was the latest as of the time of this writing, so make sure you’re getting the latest version.  (Note: it looks like the link below gets cut off, so here’s a link to the NConf download page where you can grab the latest source)

wget http://sourceforge.net/projects/nconf/files/nconf/1.2.6-0/nconf-1.2.6-0.tgz/download
Unpack it to your webroot:
sudo tar xzvf nconf-1.2.6-0.tgz -C /var/www
Change the owner of the folder and files you just extracted to the web user:
sudo chown -R www-data:www-data /var/www/nconf
Now you’ll need to create a MySQL database to hold the NConf configuration data.  Using phpMyAdmin (sudo apt-get install phpmyadmin if you don’t have it) do the following:
  • Privileges -> Add a new User
  • User name: nconf
  • Host: localhost
  • Password: (generate, and make a note of the generated password)
  • Create database with the same name and grant all privileges
  • Go
  • Privileges -> reload the privileges

Then browse to http://<yourserver>/nconf and follow the prompts to finish initial configuration of NConf.

When you get to the database configuration page, enter nconf for the username and the database name, and use the generated password you should have made a note of earlier. For most everything else, accept the defaults, but change the NAGIOS_BIN variable to /usr/sbin/nagios3 to reference the right location.

Finally, remove the folders and files referenced at the end of the installation process:

sudo rm -r /var/www/nconf/INSTALL
sudo rm /var/www/nconf/INSTALL.php
sudo rm -r /var/www/nconf/UPDATE
sudo rm /var/www/nconf/UPDATE.php

At this point, you should be able to login to NConf, although it won’t be doing anything of importance.

Configuring Nagios to use NConf

We’ll need to make some changes to one of the Nagios configuration files, so (using sudo) open up /etc/nagios3/nagios.cfg in your favorite editor and delete or comment out all the lines that begin with cfg_dir= or cfg_file= and add the following lines:

cfg_dir=/etc/nagios3/global
cfg_dir=/etc/nagios3/Default_collector

Back at the terminal, run the following command to create a folder for NConf to dump the configuration files it generates.

sudo mkdir /etc/nagios3/import

Configuring NConf to Deploy Nagios Configurations Automatically

Almost there. Using sudo, open up /var/www/nconf/ADD-ONS/deploy_local.sh and make the following changes to paths:

OUTPUT_DIR="/var/www/nconf/output/"
NAGIOS_DIR="/etc/nagios3/"
...
/etc/init.d/nagios3 reload

GM and Allison pointed out in the comments that they had to set the execute bit on this file:

chmod +x /var/www/nconf/ADD-ONS/deploy_local.sh

This script will deploy the generated configuration package and then reload the running instance of Nagios, but it’s easiest to use just installed in the root crontab.

sudo crontab -e

and adding the line

* * * * * /var/www/nconf/ADD-ONS/deploy_local.sh

Final Steps

After saving and closing the root crontab, log back into NConf and take a look around.  You’ll see some sample definitions and some predefined services for the localhost computer.  You may want to delete the check_local_mrtgtraf and check_local_procs services, as the first one doesn’t work without additional configuration and the second one is a sample definition, but you can make those changes at your leisure.

Once you’re ready, click Generate Nagios config, and if all goes well, you’ll see something like the following:

[ Initializing NConf perl-API (library version 0.2, written by A. Gargiulo) ]
[ Copyright (c) 2006-2009 Sunrise Communications AG, Zurich, Switzerland    ]

[INFO]  Starting generate_config script
[INFO]  Generating global config files
[INFO]  Generating config for Nagios-collector 'Default Nagios'
[INFO]  Ended generate_config script

Running syntax check:

	  Default_collector:	 Total Warnings: 0  Total Errors: 0
Changes updated successfully.

Now log back into Nagios and click on Service Detail.  Within a minute or two, you should see the hosts and services change to reference the configuration as generated from NConf.

If that works, then huzzah!

Start making your configuration changes in NConf and enjoy not having the manipulate those Nagios conf files by hand anymore!

Troubleshooting, Tips and Caveats

Try running the deploy_local.sh script by hand (with sudo) if it doesn’t appear that Nagios is getting the configurations from NConf. You may be able to glean some information from the output of that script.

New hosts not showing up in Nagios after being created in NConf? Make sure you’ve selected Default Nagios under “monitored by” when defining the host, or that host won’t get assigned properly.

This setup process effectively disables all the command definitions provided by the package install of Nagios, which are stored under /etc/nagios-plugins/config.  I looked through them and compared them to the ones provided by default with NConf, and I was fine with what NConf provided.  NConf provides a mechanism to import command definitions if you really find that you need them.

I have no idea how this setup will hold up under an upgrade of the Nagios package.  We’ll see when the time comes, and if I remember, I’ll update these notes.

If you have any trouble with the steps provided above, please comment and I’ll do what I can to assist.

More Information

Take a look at the following documentation from NConf:

Categories
Software

Quick and Dirty (Free)DOS Bootable USB Drive Under Linux

I found myself needing to flash a BIOS today, which required a DOS or Windows 98 bootable floppy disk.  Forget that.  Let’s use a USB drive instead!

Googling around returned methods that seemed way too involved for what I was trying to do.  Here’s the three commands I ended up using:

wget http://www.fdos.org/bootdisks/autogen/FDOEM.144.gz
gunzip FDOEM.144.gz
sudo dd if=FDOEM.144 of=/dev/sdb

Be sure to change /dev/sdb to whatever your USB drive shows up as. (Disclaimer: I’m not responsible if you end up nuking your secondary hard drive)

Then unplug the drive and plug it back in.  Unzip your files onto the drive (mine shows up as /media/FREEDOS), eject, and flash away!

You’ll end up with about 1.3 MB of free space on the drive, which should be enough for the majority of things.

Categories
Software

Linux ‘rename’ utility on OS X

(edit 12/11/2012 to add homebrew alternative)

So you you want the powerful Perl-based rename utility that comes with many Linux and *BSD distributions, but sadly not on Apple’s mighty OS X? Well so did I. (Go away you fanboys of all types. I hear not your rants about which operating system is for the win)

Thinking about it for a moment, I realize there’s nothing really stopping me from just taking the script, and thankfully it’s free software and we can pretty much do whatever we need to with it.

The rename utility is tucked away under /usr/bin/ on my Ubuntu machine.  To get it to your OS X machine, it’s just a matter of copying /usr/bin/rename on a Linux box and dropping it in /usr/local/bin/ or somewhere else that’s in your PATH on your OS X box.  If you’re really feeling the need for the man page, you can grab it from /usr/share/man/man1/rename.1.gz.  I stuck the man page in /usr/local/share/man/man1/.

And that’s it! You can now go about using Perl regular expressions to rename multiple files and get lost in everything that is regexp.

Don’t have access to a Linux machine, you say? Oh fine. Here’s what I pulled from my Ubuntu 9.04 distro:

If you download the rename utility from here, you’ll probably need to run ‘chmod a+x’ to give execute permissions to the file after copying it to the appropriate place but otherwise things should be the same as above.

Examples

Wondering how it works?  Say you’d like to rename a bunch of .wav files and prepend the phrase ‘iheartrobots’ to the front of each filename (practical, I know).  Well you’d type the following:

rename -v 's/^/iheartrobots/' *.wav

The ‘-v’ parameter prints the names of the files that are being renamed.  Helpful and I use it most of the time.

Or how about replacing all instances of the word ‘cat’ with the word ‘dog’ in the filename of all .rtf files in a folder? Bam.

rename -v 's/cat/dog/i' *.wav

The ‘i’ at the end of the regular expression makes the search case-insensitive.

Finally, say you had a bunch of files that looked like this: Waltzing.With.Famous.People.S01E09.HDTV.XviD-SYS.[bztv].avi

And you wanted to get rid of everything between S01E09 and .avi, you could use the following:

rename -v 's/HDTV.*avi/avi/i' *.avi

The Homebrew Alternative

Now, thanks to Homebrew, you can skip all this nonsense and just do:

brew install rename

If you’re not using Homebrew, you should be.

More Information

If you’d like to get more information on the rename utility described here, the most obvious place is the man page (man rename) or online here.

Additionally, since it’s a Perl utility, more information on the Perl regular expression pattern-matching language can be found straight from the horse’s mouth.  Regular expressions are a madness unto themselves, but once you get the hang of it, most powerful stuff.