Category: System Administration

Categories
System Administration

Adding Rails log rotation to Dreamhost (compiling your own logrotate)

(credit goes to Joey Geiger for the original Gist)

Trying to use logrotate to rotate your Rails logs on Dreamhost?   You’ll need to roll your own copy of it.  Fortunately, the process isn’t too bad.  Let’s jump right into it.

SSH to your host and run the following commands (read my notes below if you’re not super gusty):

# create a new cron job (crontab -e) or use panel.dreamhost.com and run it daily
0 0 * * * /home/<your_account>/opt/bin/logrotate -f -s /home/<your_account>/opt/lib/logrotate.status /home/<your_account>/opt/etc/logrotate.conf
cd ~
mkdir -p opt/bin
mkdir opt/lib
mkdir opt/etc
mkdir src
cd src
wget http://www.sfr-fresh.com/unix/privat/logrotate-3.7.9.tar.gz
wget http://www.sfr-fresh.com/linux/misc/popt-1.16.tar.gz
tar xzvf logrotate-3.7.9.tar.gz
tar xzvf popt-1.16.tar.gz
cd popt-1.16
./configure --prefix=/home/<your_account>/opt/
make
make install
cd ../logrotate-3.7.9
# the first make will fail on the final compile (with 'cannot find -lpopt' message)
# some include problem, but this works so i'm not inclined to care too much
make POPT_DIR=/home/<your_account>/opt/include
# this make should complete
make POPT_DIR=/home/<your_account>/opt/lib
cp logrotate ~/opt/bin/
cd ~/opt/etc
vi logrotate.conf
# paste in the logrotate.conf file
/home/<your_account>/<your_website>/shared/log/*.log {
  daily
  missingok
  rotate 7
  compress
  delaycompress
  notifempty
  copytruncate
}


A couple of quick notes here:

    The most substantial difference from jgeiger’s post is the compiling of the popt library, so that in turn, logrotate would compile.  I really don’t know what’s messed up when you try to link to popt (which leads to the compile failing the first time around), but the workaround seems to work just fine.  If someone happens to know what magic flags are necessary to get the compile to work smoothly the first time, please fork the notes and let me know.
    Joe’s Own Editor was the default editor when I tried to edit my crontab, and it makes my brain hurt.  Edit your .bash_profile and add export EDITOR="/usr/bin/vim" to use vim instead. Logout and back in for the change to take effect.

(disclaimer: Don’t break your stuff. Or Dreamhost’s stuff.)

Categories
System Administration

Nagios and NConf on Ubuntu 10.04 Lucid Lynx

As of December 2011, NConf has released version 1.3.0 which changes the deployment methods for configs. Take a look at my NConf 1.3.0 upgrade guide!

Have you been wondering about how to get Nagios and NConf setup on your Ubuntu server?  Well, I took some notes the last time around, and here they are, for anyone that may find them useful.  Here we go!

(disclaimer: don’t screw up your system.)

Prerequisites

If you didn’t check the box to configure your box as a LAMP (Linux, Apache, MySQL, PHP) server when you first setup your box, do that first with:

sudo apt-get install apache2
sudo apt-get install mysql-server mysql-client
sudo apt-get install php5 libapache2-mod-php5

If you want phpMyAdmin (for web-based MySQL administration):

sudo apt-get install phpmyadmin

Installing Nagios

Install Nagios with

sudo apt-get install nagios3

It should install all the necessary dependencies. Follow the prompts to get it configured and test it out by browsing to http://<yourserver>/nagios3

If you would like to monitor Windows servers using NSClient++ (which I am doing), also install the NRPE plugin with

sudo apt-get install nagios-nrpe-plugin

I felt like I ought to backup the config files before I started messing with them, so I backed them up with

sudo cp -r /etc/nagios3 /etc/nagios3.backup

Installing NConf

NConf provides a web-based frontend for configuring Nagios.  There’s no package in the Ubuntu repositories for it, but you can download it pretty easily.  Version 1.2.6 was the latest as of the time of this writing, so make sure you’re getting the latest version.  (Note: it looks like the link below gets cut off, so here’s a link to the NConf download page where you can grab the latest source)

wget http://sourceforge.net/projects/nconf/files/nconf/1.2.6-0/nconf-1.2.6-0.tgz/download
Unpack it to your webroot:
sudo tar xzvf nconf-1.2.6-0.tgz -C /var/www
Change the owner of the folder and files you just extracted to the web user:
sudo chown -R www-data:www-data /var/www/nconf
Now you’ll need to create a MySQL database to hold the NConf configuration data.  Using phpMyAdmin (sudo apt-get install phpmyadmin if you don’t have it) do the following:
  • Privileges -> Add a new User
  • User name: nconf
  • Host: localhost
  • Password: (generate, and make a note of the generated password)
  • Create database with the same name and grant all privileges
  • Go
  • Privileges -> reload the privileges

Then browse to http://<yourserver>/nconf and follow the prompts to finish initial configuration of NConf.

When you get to the database configuration page, enter nconf for the username and the database name, and use the generated password you should have made a note of earlier. For most everything else, accept the defaults, but change the NAGIOS_BIN variable to /usr/sbin/nagios3 to reference the right location.

Finally, remove the folders and files referenced at the end of the installation process:

sudo rm -r /var/www/nconf/INSTALL
sudo rm /var/www/nconf/INSTALL.php
sudo rm -r /var/www/nconf/UPDATE
sudo rm /var/www/nconf/UPDATE.php

At this point, you should be able to login to NConf, although it won’t be doing anything of importance.

Configuring Nagios to use NConf

We’ll need to make some changes to one of the Nagios configuration files, so (using sudo) open up /etc/nagios3/nagios.cfg in your favorite editor and delete or comment out all the lines that begin with cfg_dir= or cfg_file= and add the following lines:

cfg_dir=/etc/nagios3/global
cfg_dir=/etc/nagios3/Default_collector

Back at the terminal, run the following command to create a folder for NConf to dump the configuration files it generates.

sudo mkdir /etc/nagios3/import

Configuring NConf to Deploy Nagios Configurations Automatically

Almost there. Using sudo, open up /var/www/nconf/ADD-ONS/deploy_local.sh and make the following changes to paths:

OUTPUT_DIR="/var/www/nconf/output/"
NAGIOS_DIR="/etc/nagios3/"
...
/etc/init.d/nagios3 reload

GM and Allison pointed out in the comments that they had to set the execute bit on this file:

chmod +x /var/www/nconf/ADD-ONS/deploy_local.sh

This script will deploy the generated configuration package and then reload the running instance of Nagios, but it’s easiest to use just installed in the root crontab.

sudo crontab -e

and adding the line

* * * * * /var/www/nconf/ADD-ONS/deploy_local.sh

Final Steps

After saving and closing the root crontab, log back into NConf and take a look around.  You’ll see some sample definitions and some predefined services for the localhost computer.  You may want to delete the check_local_mrtgtraf and check_local_procs services, as the first one doesn’t work without additional configuration and the second one is a sample definition, but you can make those changes at your leisure.

Once you’re ready, click Generate Nagios config, and if all goes well, you’ll see something like the following:

[ Initializing NConf perl-API (library version 0.2, written by A. Gargiulo) ]
[ Copyright (c) 2006-2009 Sunrise Communications AG, Zurich, Switzerland    ]

[INFO]  Starting generate_config script
[INFO]  Generating global config files
[INFO]  Generating config for Nagios-collector 'Default Nagios'
[INFO]  Ended generate_config script

Running syntax check:

	  Default_collector:	 Total Warnings: 0  Total Errors: 0
Changes updated successfully.

Now log back into Nagios and click on Service Detail.  Within a minute or two, you should see the hosts and services change to reference the configuration as generated from NConf.

If that works, then huzzah!

Start making your configuration changes in NConf and enjoy not having the manipulate those Nagios conf files by hand anymore!

Troubleshooting, Tips and Caveats

Try running the deploy_local.sh script by hand (with sudo) if it doesn’t appear that Nagios is getting the configurations from NConf. You may be able to glean some information from the output of that script.

New hosts not showing up in Nagios after being created in NConf? Make sure you’ve selected Default Nagios under “monitored by” when defining the host, or that host won’t get assigned properly.

This setup process effectively disables all the command definitions provided by the package install of Nagios, which are stored under /etc/nagios-plugins/config.  I looked through them and compared them to the ones provided by default with NConf, and I was fine with what NConf provided.  NConf provides a mechanism to import command definitions if you really find that you need them.

I have no idea how this setup will hold up under an upgrade of the Nagios package.  We’ll see when the time comes, and if I remember, I’ll update these notes.

If you have any trouble with the steps provided above, please comment and I’ll do what I can to assist.

More Information

Take a look at the following documentation from NConf:

Categories
System Administration

Forwarding Yahoo! Mail to Gmail (For Free!)

I’ve got way too many email addresses. Free email accounts opened up as a teenager that I don’t want to get rid of because they hold some strange nostalgic value. Hotmail, Gmail, Yahoo…I’ve even got an account with RocketMail, which eventually became Yahoo! Mail.

So consolidation is my other option.  I’m using Google Apps for Domains which essentially gives me Gmail on my own domain.  Most of the free email services have some mechanism which I can use to forward to my Gmail account, but Yahoo! requires their paid service.  No fun.

If I was using a desktop email client I could just implement YPOPs! or FreePOPs. But I’m not.

Linux to the rescue! Using Exim and FetchYahoo I can now periodically check my Yahoo! Mail accounts and redirect them to my Gmail account. FetchYahoo is great because it provides the mechanism to forward emails to another email address; the products mentioned about don’t appear to have this feature.

Here’s how I did it, using Ubuntu 8.10 (Intrepid Ibex).

Reconfigure Exim4 to use Gmail SMTP (Note: POP must be enabled in your Gmail settings). I used the instructions I found on TechSutra, which I’m not going to reproduce here because I’m lazy.

Remember to comment out or remove any other smarthost defined with “domains = ! +local_domains” as well as any other authenticators where “public_name = LOGIN”. These were both defined for me and I originally missed these steps.

Ensure that Exim4 is working. See step 3 from TechSutra’s instructions.

Download and install FetchYahoo (2.13.3 as of this writing). Alternatively, install from the repos (it’s an older version though):

sudo apt-get install fetchyahoo

Test FetchYahoo. Replace <username> and <password> with your, well, username and password.

fetchyahoo --mailhost=localhost --username=<username>@yahoo.com \
--password=<password> --onlylistmessages

You should see a list of the emails in your inbox. If it bombs out, you might try logging into your Yahoo! Mail and changing the interface from Classic to New and vice-versa.

Try forwarding to Gmail using FetchYahoo. You can limit the number of emails it fetches by using –maxmessages=N where N is the maximum number of messages to grab, but I only had a handful of emails in my inbox at the time.  Once the emails are processed, they’ll end up in the Trash, so if there are problems, you can just move them back to the inbox and try again.

fetchyahoo --sendto=<username>@gmail.com --mailhost=localhost \
--username=<username>@yahoo.com --password=<password>

You should see something like the following:

Only forwarding e-mail, local delivery turned off.
Logging in securely via SSL as <username>@yahoo.com on Wed Apr 15 00:04:15 2009
Country code : us    FetchYahoo! Version: 2.13.3
Successfully logged in as <username>@yahoo.com.
Marking messages read on the server

Fetching mail from folder: Inbox
Getting Message ID(s) for message(s) 1 - 5.
Got 5 Message IDs
._._.__.5
Finished downloading 5 messages.
5 message(s) have been deleted.

You can monitor the Exim4 logfile to see if the emails are being sent properly:

tail -f /var/log/exim/mainlog

Check your Gmail account for the forwarded emails. Now the only glitch I’m having here is that forwarded emails seem to be coming into All Mail rather than Inbox and are already marked as read.  If anyone has any ideas on this, please let me know.

If all is well at this point, then congratulations! If all is not well, then I don’t know what to tell you, slick. It worked for me.

Create FetchYahoo configuration file. FetchYahoo allows for an /etc/fetchyahoorc file or ~/.fetchyahoorc file rather than passing all the options via the command line. From the FetchYahoo source folder, copy fetchyahoorc to your home directory:

cp fetchyahoorc ~/.fetchyahoorc

Then edit the file, substituting your own values as applicable.  The most important values are:

username = yahoo-user-name@yahoo.com
password = yahoo-password
use-forward = 1
mail-host = localhost
send-to = gmail-user-name@gmail.com

If you’d rather store a hashed password than the plaintext, you can run

fetchyahoo --md5hex

which will give you a md5 hash of your password suitable for storing in the configuration file.

Save and test the config file by running fetchyahoo with no parameters.

Schedule to run periodically via cron or use the daemon mode. I’m using cron because I’ll probably only check these accounts every few days and I don’t need the process just hanging around in the meantime.  The –repeat-interval=N command line parameter (or just repeat-interval = N in the config file) is all you need to run the process as a daemon.

And that’s it!  So until the Yahoo! folks decide to come around and allow for POP3 access or some other mechanism to get email out from their stronghold, this is what I’ll use to get my coveted bits of spam to Gmail.

Update: The original procedure described above ended up dropping forwarded emails from Yahoo into the All Mail folder rather than the Inbox due to the fact that I was sending to and from the same Gmail account.  Since I implemented this procedure using Google Apps for Domains, I simply created a generic account from which I am sending emails, and all is now well with the world.

Resources: