Categories
System Administration

Adding Rails log rotation to Dreamhost (compiling your own logrotate)

(credit goes to Joey Geiger for the original Gist)

Trying to use logrotate to rotate your Rails logs on Dreamhost?   You’ll need to roll your own copy of it.  Fortunately, the process isn’t too bad.  Let’s jump right into it.

SSH to your host and run the following commands (read my notes below if you’re not super gusty):

# create a new cron job (crontab -e) or use panel.dreamhost.com and run it daily
0 0 * * * /home/<your_account>/opt/bin/logrotate -f -s /home/<your_account>/opt/lib/logrotate.status /home/<your_account>/opt/etc/logrotate.conf
cd ~
mkdir -p opt/bin
mkdir opt/lib
mkdir opt/etc
mkdir src
cd src
wget http://www.sfr-fresh.com/unix/privat/logrotate-3.7.9.tar.gz
wget http://www.sfr-fresh.com/linux/misc/popt-1.16.tar.gz
tar xzvf logrotate-3.7.9.tar.gz
tar xzvf popt-1.16.tar.gz
cd popt-1.16
./configure --prefix=/home/<your_account>/opt/
make
make install
cd ../logrotate-3.7.9
# the first make will fail on the final compile (with 'cannot find -lpopt' message)
# some include problem, but this works so i'm not inclined to care too much
make POPT_DIR=/home/<your_account>/opt/include
# this make should complete
make POPT_DIR=/home/<your_account>/opt/lib
cp logrotate ~/opt/bin/
cd ~/opt/etc
vi logrotate.conf
# paste in the logrotate.conf file
/home/<your_account>/<your_website>/shared/log/*.log {
  daily
  missingok
  rotate 7
  compress
  delaycompress
  notifempty
  copytruncate
}


A couple of quick notes here:

    The most substantial difference from jgeiger’s post is the compiling of the popt library, so that in turn, logrotate would compile.  I really don’t know what’s messed up when you try to link to popt (which leads to the compile failing the first time around), but the workaround seems to work just fine.  If someone happens to know what magic flags are necessary to get the compile to work smoothly the first time, please fork the notes and let me know.
    Joe’s Own Editor was the default editor when I tried to edit my crontab, and it makes my brain hurt.  Edit your .bash_profile and add export EDITOR="/usr/bin/vim" to use vim instead. Logout and back in for the change to take effect.

(disclaimer: Don’t break your stuff. Or Dreamhost’s stuff.)

Categories
System Administration

Nagios and NConf on Ubuntu 10.04 Lucid Lynx

As of December 2011, NConf has released version 1.3.0 which changes the deployment methods for configs. Take a look at my NConf 1.3.0 upgrade guide!

Have you been wondering about how to get Nagios and NConf setup on your Ubuntu server?  Well, I took some notes the last time around, and here they are, for anyone that may find them useful.  Here we go!

(disclaimer: don’t screw up your system.)

Prerequisites

If you didn’t check the box to configure your box as a LAMP (Linux, Apache, MySQL, PHP) server when you first setup your box, do that first with:

sudo apt-get install apache2
sudo apt-get install mysql-server mysql-client
sudo apt-get install php5 libapache2-mod-php5

If you want phpMyAdmin (for web-based MySQL administration):

sudo apt-get install phpmyadmin

Installing Nagios

Install Nagios with

sudo apt-get install nagios3

It should install all the necessary dependencies. Follow the prompts to get it configured and test it out by browsing to http://<yourserver>/nagios3

If you would like to monitor Windows servers using NSClient++ (which I am doing), also install the NRPE plugin with

sudo apt-get install nagios-nrpe-plugin

I felt like I ought to backup the config files before I started messing with them, so I backed them up with

sudo cp -r /etc/nagios3 /etc/nagios3.backup

Installing NConf

NConf provides a web-based frontend for configuring Nagios.  There’s no package in the Ubuntu repositories for it, but you can download it pretty easily.  Version 1.2.6 was the latest as of the time of this writing, so make sure you’re getting the latest version.  (Note: it looks like the link below gets cut off, so here’s a link to the NConf download page where you can grab the latest source)

wget http://sourceforge.net/projects/nconf/files/nconf/1.2.6-0/nconf-1.2.6-0.tgz/download
Unpack it to your webroot:
sudo tar xzvf nconf-1.2.6-0.tgz -C /var/www
Change the owner of the folder and files you just extracted to the web user:
sudo chown -R www-data:www-data /var/www/nconf
Now you’ll need to create a MySQL database to hold the NConf configuration data.  Using phpMyAdmin (sudo apt-get install phpmyadmin if you don’t have it) do the following:
  • Privileges -> Add a new User
  • User name: nconf
  • Host: localhost
  • Password: (generate, and make a note of the generated password)
  • Create database with the same name and grant all privileges
  • Go
  • Privileges -> reload the privileges

Then browse to http://<yourserver>/nconf and follow the prompts to finish initial configuration of NConf.

When you get to the database configuration page, enter nconf for the username and the database name, and use the generated password you should have made a note of earlier. For most everything else, accept the defaults, but change the NAGIOS_BIN variable to /usr/sbin/nagios3 to reference the right location.

Finally, remove the folders and files referenced at the end of the installation process:

sudo rm -r /var/www/nconf/INSTALL
sudo rm /var/www/nconf/INSTALL.php
sudo rm -r /var/www/nconf/UPDATE
sudo rm /var/www/nconf/UPDATE.php

At this point, you should be able to login to NConf, although it won’t be doing anything of importance.

Configuring Nagios to use NConf

We’ll need to make some changes to one of the Nagios configuration files, so (using sudo) open up /etc/nagios3/nagios.cfg in your favorite editor and delete or comment out all the lines that begin with cfg_dir= or cfg_file= and add the following lines:

cfg_dir=/etc/nagios3/global
cfg_dir=/etc/nagios3/Default_collector

Back at the terminal, run the following command to create a folder for NConf to dump the configuration files it generates.

sudo mkdir /etc/nagios3/import

Configuring NConf to Deploy Nagios Configurations Automatically

Almost there. Using sudo, open up /var/www/nconf/ADD-ONS/deploy_local.sh and make the following changes to paths:

OUTPUT_DIR="/var/www/nconf/output/"
NAGIOS_DIR="/etc/nagios3/"
...
/etc/init.d/nagios3 reload

GM and Allison pointed out in the comments that they had to set the execute bit on this file:

chmod +x /var/www/nconf/ADD-ONS/deploy_local.sh

This script will deploy the generated configuration package and then reload the running instance of Nagios, but it’s easiest to use just installed in the root crontab.

sudo crontab -e

and adding the line

* * * * * /var/www/nconf/ADD-ONS/deploy_local.sh

Final Steps

After saving and closing the root crontab, log back into NConf and take a look around.  You’ll see some sample definitions and some predefined services for the localhost computer.  You may want to delete the check_local_mrtgtraf and check_local_procs services, as the first one doesn’t work without additional configuration and the second one is a sample definition, but you can make those changes at your leisure.

Once you’re ready, click Generate Nagios config, and if all goes well, you’ll see something like the following:

[ Initializing NConf perl-API (library version 0.2, written by A. Gargiulo) ]
[ Copyright (c) 2006-2009 Sunrise Communications AG, Zurich, Switzerland    ]

[INFO]  Starting generate_config script
[INFO]  Generating global config files
[INFO]  Generating config for Nagios-collector 'Default Nagios'
[INFO]  Ended generate_config script

Running syntax check:

	  Default_collector:	 Total Warnings: 0  Total Errors: 0
Changes updated successfully.

Now log back into Nagios and click on Service Detail.  Within a minute or two, you should see the hosts and services change to reference the configuration as generated from NConf.

If that works, then huzzah!

Start making your configuration changes in NConf and enjoy not having the manipulate those Nagios conf files by hand anymore!

Troubleshooting, Tips and Caveats

Try running the deploy_local.sh script by hand (with sudo) if it doesn’t appear that Nagios is getting the configurations from NConf. You may be able to glean some information from the output of that script.

New hosts not showing up in Nagios after being created in NConf? Make sure you’ve selected Default Nagios under “monitored by” when defining the host, or that host won’t get assigned properly.

This setup process effectively disables all the command definitions provided by the package install of Nagios, which are stored under /etc/nagios-plugins/config.  I looked through them and compared them to the ones provided by default with NConf, and I was fine with what NConf provided.  NConf provides a mechanism to import command definitions if you really find that you need them.

I have no idea how this setup will hold up under an upgrade of the Nagios package.  We’ll see when the time comes, and if I remember, I’ll update these notes.

If you have any trouble with the steps provided above, please comment and I’ll do what I can to assist.

More Information

Take a look at the following documentation from NConf:

Categories
Photography

Typography and the Ag Museum

So I got a new lens for my Nikon (a 50mm f/1.8D AF) and boy is it a lot of fun.  The D60 body doesn’t have an in-body autofocus motor, which is probably one of reasons why it’s so small, but it’s been a pleasant challenge forcing myself to manually focus with this lens.

These are a few of the resulting shots, taken at my local agricultural museum.

Super Chief

Armall

Regular

Bombardier Limitee

View the full set on flickr.

Categories
Uncategorized

Star Wars Mafia

Mafia (or Werewolf or Assassin) is one of those party games that almost inevitably arises when there is a good group of more than five people (probably tending towards the geeky) get together with not a whole lot to do.  It’s a socially engaging game that can easily burn up an hour or two of an evening.

If you’ve never played, check out the Wikipedia article on Mafia to get some background information, or this Wikihow article on how to play.

The following is one of the many variants of Mafia, which happens to use everybody’s favorite Star Wars characters instead of the mafia or werewolves.  It was introduced to me by Peter and David Lindstrom, and the information here is more or less verbatim as I got it from them.  I’m posting it as a reference to myself and anyone else who might be interested to play this version of Mafia.

If anyone has any questions about the Star Wars Mafia variant, please post them in the comments and if I can’t provide insight, I’ll get a hold of Peter and David and find out from them. Enjoy!

The Force is the equivalent of the Narrator or Moderator in the standard Mafia game. Depending on the size of the group, the last few characters on the Rebel Alliance side may not be used.

The Rebel Alliance

  • Jedi Knight—Once per night the Jedi knight may either question or challenge. To question, point to a person. 10 challenge, make a sign like you are fighting with a light saber, and then point to a person. If the Jedi Knight elects to question, the Force will tell him whether the person he indicated is good or bad. A thumbs up means good. Thumbs down means bad. If the Jedi Knight elects to challenge a character, he will win if the character is bad. The bad character will be dead the next morning. However, if the Jedi Knight challenges a good character, he will die.
  • Obi Wan Kenobi—Can talk after he is dead.
  • Chewbacca—Once per game Chewbacca may interrupt discussions or a trial and eat anyone he chooses. If he tries to eat Han, Han may reveal his card and say “Will somebody get this big walking carpet out of my way?” Chewy can then choose to eat him or not. Night falls immediately thereafter.
  • Thermal Detonator—If the thermal detonator is killed during the day, he kills the person to his left. If he is killed at night. he kills the first Sith to his left.
  • Lando Calrisian—Lando may elect to smuggle once per game, during the night. If he elects to smuggle, the Force will exchange the identities of two people during the night. Only those people whose identities have changed will be aware of the switch.
  • Princess Leia—If Princess Leia is on trial, she may reveal her card, and say “Will somebody get this big walking carpet out of my way?” Han Must then reveal his card and save her.
  • Han Solo—Must save Princess Leia if she asks. If Chewy tries to eat him, he may reveal his card and say “Chewy, you stupid walking carpet.”
  • Bounty Hunter (I know, he’s not really a Rebel, but for the sake of the game he’s on this side)—Picks one person during each night to be his death buddy. If the bounty hunter dies during the night, his death buddy also dies.
  • Yoda—Raises somebody from the dead once per night.
  • Owen & Beru Lars—Die the first night.
  • Salacious Crumb—Must vote as Jabba votes.
  • Jabba the Hut—Controls Salacious Crumb’s vote
  • Jar Jar Binks—Once his identity is revealed by the probe droid, he must speak in a fake Jamaican accent. Once per night he points to someone else. Then they also must speak with an accent.

The Dark Side

  • Darth Vader—Conspires with the Emperor to kill one person each night.
  • The Emperor—Conspires with Darth Vader to kill one person each night.
  • Apprentice—Takes over for the Emperor if the Emperor dies.
  • Probe Droid—Reveals one identity per night. Everyone knows what character has been revealed, but only the probe droid knows which person has that identity.
  • Storm Trooper—Can never die. He just keeps reappearing. He doesn’t need to be killed for the good side to win.

First Night

  • Salacious Crumb and Jabba see each other.
  • Darth Vader and the Emperor see each other.
  • Darth Vader, the Emperor, and the Probe Droid see each other.
  • The Apprentice and the Emperor see each other.

Every Night

  • Yoda may bring someone back to life.
  • Lando may smuggle (if he has not already done so)
  • Jedi Knight may question or challenge
  • Bounty Hunter picks his death buddy
  • If Jar Jar has been revealed, he picks somebody to speak fake Jamaican
  • The Sith kill somebody
  • Probe Droid reveals an identity
Categories
Software

Quick and Dirty (Free)DOS Bootable USB Drive Under Linux

I found myself needing to flash a BIOS today, which required a DOS or Windows 98 bootable floppy disk.  Forget that.  Let’s use a USB drive instead!

Googling around returned methods that seemed way too involved for what I was trying to do.  Here’s the three commands I ended up using:

wget http://www.fdos.org/bootdisks/autogen/FDOEM.144.gz
gunzip FDOEM.144.gz
sudo dd if=FDOEM.144 of=/dev/sdb

Be sure to change /dev/sdb to whatever your USB drive shows up as. (Disclaimer: I’m not responsible if you end up nuking your secondary hard drive)

Then unplug the drive and plug it back in.  Unzip your files onto the drive (mine shows up as /media/FREEDOS), eject, and flash away!

You’ll end up with about 1.3 MB of free space on the drive, which should be enough for the majority of things.

Categories
Software

Linux ‘rename’ utility on OS X

(edit 12/11/2012 to add homebrew alternative)

So you you want the powerful Perl-based rename utility that comes with many Linux and *BSD distributions, but sadly not on Apple’s mighty OS X? Well so did I. (Go away you fanboys of all types. I hear not your rants about which operating system is for the win)

Thinking about it for a moment, I realize there’s nothing really stopping me from just taking the script, and thankfully it’s free software and we can pretty much do whatever we need to with it.

The rename utility is tucked away under /usr/bin/ on my Ubuntu machine.  To get it to your OS X machine, it’s just a matter of copying /usr/bin/rename on a Linux box and dropping it in /usr/local/bin/ or somewhere else that’s in your PATH on your OS X box.  If you’re really feeling the need for the man page, you can grab it from /usr/share/man/man1/rename.1.gz.  I stuck the man page in /usr/local/share/man/man1/.

And that’s it! You can now go about using Perl regular expressions to rename multiple files and get lost in everything that is regexp.

Don’t have access to a Linux machine, you say? Oh fine. Here’s what I pulled from my Ubuntu 9.04 distro:

If you download the rename utility from here, you’ll probably need to run ‘chmod a+x’ to give execute permissions to the file after copying it to the appropriate place but otherwise things should be the same as above.

Examples

Wondering how it works?  Say you’d like to rename a bunch of .wav files and prepend the phrase ‘iheartrobots’ to the front of each filename (practical, I know).  Well you’d type the following:

rename -v 's/^/iheartrobots/' *.wav

The ‘-v’ parameter prints the names of the files that are being renamed.  Helpful and I use it most of the time.

Or how about replacing all instances of the word ‘cat’ with the word ‘dog’ in the filename of all .rtf files in a folder? Bam.

rename -v 's/cat/dog/i' *.wav

The ‘i’ at the end of the regular expression makes the search case-insensitive.

Finally, say you had a bunch of files that looked like this: Waltzing.With.Famous.People.S01E09.HDTV.XviD-SYS.[bztv].avi

And you wanted to get rid of everything between S01E09 and .avi, you could use the following:

rename -v 's/HDTV.*avi/avi/i' *.avi

The Homebrew Alternative

Now, thanks to Homebrew, you can skip all this nonsense and just do:

brew install rename

If you’re not using Homebrew, you should be.

More Information

If you’d like to get more information on the rename utility described here, the most obvious place is the man page (man rename) or online here.

Additionally, since it’s a Perl utility, more information on the Perl regular expression pattern-matching language can be found straight from the horse’s mouth.  Regular expressions are a madness unto themselves, but once you get the hang of it, most powerful stuff.

Categories
Software

The SoX of Silence

SoX is, by their own definition, the Swiss Army knife of audio manipulation.

And no doubt it’s full of fun with slicing and dicing and playback and recording and filtering and effects capabilities.

But SoX is a command line tool, which means obscure syntax and parameters in order to get things done.

I’ve been trying off and on for months to try to understand the silence filter from within SoX, which allows one to remove silence from the beginning, middle, or end of the audio. Sounds, simple, doesn’t it?  Well, it should be.

Below is the man page for the silence filter:

silence [-l] above-periods [duration threshold[d|%] [below-periods duration threshold[d|%]]

Removes silence from the beginning, middle, or end of the audio. Silence is anything below a specified threshold.

The above-periods value is used to indicate if audio should be trimmed at the beginning of the audio. A value of zero indicates no silence should be trimmed from the beginning. When specifying an non-zero above-periods, it trims audio up until it finds non-silence. Normally, when trimming silence from beginning of audio the above-periods will be 1 but it can be increased to higher values to trim all audio up to a specific count of non-silence periods. For example, if you had an audio file with two songs that each contained 2 seconds of silence before the song, you could specify an above-period of 2 to strip out both silence periods and the first song.

When above-periods is non-zero, you must also specify a duration and threshold. Duration indications the amount of time that non-silence must be detected before it stops trimming audio. By increasing the duration, burst of noise can be treated as silence and trimmed off.

Threshold is used to indicate what sample value you should treat as silence. For digital audio, a value of 0 may be fine but for audio recorded from analog, you may wish to increase the value to account for background noise.

When optionally trimming silence from the end of the audio, you specify a below-periods count. In this case, below-period means to remove all audio after silence is detected. Normally, this will be a value 1 of but it can be increased to skip over periods of silence that are wanted. For example, if you have a song with 2 seconds of silence in the middle and 2 second at the end, you could set below-period to a value of 2 to skip over the silence in the middle of the audio.

For below-periods, duration specifies a period of silence that must exist before audio is not copied any more. By specifying a higher duration, silence that is wanted can be left in the audio. For example, if you have a song with an expected 1 second of silence in the middle and 2 seconds of silence at the end, a duration of 2 seconds could be used to skip over the middle silence.

Unfortunately, you must know the length of the silence at the end of your audio file to trim off silence reliably. A work around is to use the silence effect in combination with the reverse effect. By first reversing the audio, you can use the above-periods to reliably trim all audio from what looks like the front of the file. Then reverse the file again to get back to normal.

To remove silence from the middle of a file, specify a below-periods that is negative. This value is then treated as a positive value and is also used to indicate the effect should restart processing as specified by the above-periods, making it suitable for removing periods of silence in the middle of the audio.

The option -l indicates that below-periods duration length of audio should be left intact at the beginning of each period of silence. For example, if you want to remove long pauses between words but do not want to remove the pauses completely.

The period counts are in units of samples. Duration counts may be in the format of hh:mm:ss.frac, or the exact count of samples. Threshold numbers may be suffixed with d to indicate the value is in decibels, or % to indicate a percentage of maximum value of the sample value (0% specifies pure digital silence).

The following example shows how this effect can be used to start a recording that does not contain the delay at the start which usually occurs between `pressing the record button’ and the start of the performance:

rec parameters filename other-effects silence 1 5 2%

Huh?

So lets try to clarify some of the mess from the man page.  First a couple of important notes:

  • When specifying duration, use a trailing zero for whole numbers of seconds (ie, 1.0 instead of 1 to specify 1 second). If you don’t, SoX assumes you’re specifying a number of samples.  Who on earth would want to specify samples instead seconds? You got me. Alternatively, you can specify durations of time in the format hh:mm:ss.frac.
  • Use at 0.1% at a minimum for an audio threshold. Even though 0% is supposed to be pure digital silence, with my test file I couldn’t get silence to trim unless I used a threshold larger than 0%. If you’d like, you can specify the threshold in decibels using d (such as -96d or -55d).
  • The realistic values for the above-period parameter are 0 and 1 and values for the below-period parameter are pretty much just -1 and 1. The documentation states that values larger than 1 can be used, but it only really makes sense for files with consistent audio breaks. Just trust me, it’s weird. I’ll get into what those values actually mean in the examples.

Now onto some examples! I’ll be showing you visually what happens to a sound file when we apply the various parameters to the silence filter.

I generated a test sound file with 60 seconds of white noise and then silenced various parts of the clip, leaving me with an audio file that looks like this:

SoX Silence Example (Original File)

Example 1: Trimming silence at the beginning

sox in.wav out1.wav silence 1 0.1 1%

The above-period parameter is first after the silence parameter, and for the sake of this article, it should be set to 1 if you want to use the filter. This example roughly translates to: trim silence (anything less than 1% volume) until we encounter sound lasting more than 0.1 seconds in duration. The output of this command produces the following:

sox in.wav out1.wav silence 1 0.1 1%

We’ve lopped off the silence at the beginning of the clip. For simplicity’s sake, we’ll refer to the 1% threshold as silence from now on.

Example 2: Ignoring noise bursts

sox in.wav out2.wav silence 1 0.3 1%

By changing the duration parameter to 0.3, we tell SoX to ignore the burst of noise at the beginning of the example clip. This produces the following:

sox in.wav out2.wav silence 1 0.3 1%

We can ignore short pops and clicks in audio by adjusting this duration parameter.

Example 3: Stopping recording when no sound detected

sox in.wav out3.wav silence 1 0.3 1% 1 0.3 1%

Now we introduce the below-period parameter it’s respective sub-parameters.  Just like the above-period parameter, just set it to 1 and call it good.  The command above translates to: trim silence until we detect at least 0.3 seconds of noise, and then trim everything after we detect at least 0.3 seconds of silence.

sox in.wav out3.wav silence 1 0.3 1% 1 0.3 1%

This returns a file with just the first 4 seconds of noise (note that we ignore that 0.25 sec burst of noise at the beginning). Where’s the rest of the clip?  Well, it’s gone. Not super practical for post-production of audio, but can be useful when recording live audio, so that SoX stops when it doesn’t encounter sound for a certain number of seconds.

So an aside: if you’re looking to trim silence from the beginning and the end of a audio file, you’ll need to utilize the reverse filter and a temp file like so:

sox in.wav temp.wav silence 1 0.1 1% reverse
sox temp.wav out.wav silence 1 0.1 1% reverse

Don’t forget to delete that temp.wav file when you’re done.

Jakob points out in the comments that you can trim silence from both ends in one fell swoop by chaining the effects like so:

sox in.wav out.wav silence 1 0.1 1% reverse silence 1 0.1 1% reverse

Example 4: Trimming all silence

sox in.wav out4.wav silence 1 0.1 1% -1 0.1 1%

By changing the below-period parameter to -1, we can trim instances of silence in the middle of the clip, by allowing the filter to restart after it detects noise of the specified duration.

sox in.wav out4.wav silence 1 0.1 1% -1 0.1 1%

In my example clip, it’s impossible to detect where the silence used to be, but with an actual podcast or other audio, it should be easier to tell.

Example 5: Ignoring short periods of silence

sox in.wav out5.wav silence 1 0.1 1% -1 0.5 1%

In similar fashion as Example 2, we can instruct SoX to ignore small moments of silence (1/2 second in this example).

sox in.wav out5.wav silence 1 0.1 1% -1 0.5 1%

When trimming silence from podcasts and the like, this prevents you from removing moments when someone stops to take a breath and making the conversation sound too rushed.

Example 6: Shortening long periods of silence

sox in.wav out6.wav silence -l 1 0.1 1% -1 2.0 1%

So what if you wanted to just shorten long moments of silence rather than remove them entirely?  Well, you need to add the -l parameter, but it needs to be placed first, before the other parameters for the filter effect. The example above results in trimming all silence longer than 2 seconds down to only 2 seconds long.

sox in.wav out6.wav silence -l 1 0.1 1% -1 2.0 1%

Note that SoX does nothing to bits of silence shorter than 2 seconds.

Example 7: Shortening long periods of silence and ignoring noise bursts

sox in.wav out7.wav silence -l 1 0.3 1% -1 2.0 1%

Finally, let’s tie it all together by trimming silence longer than 2 seconds down to 2 seconds long, but ignore noise such as pops and clicks amidst the moments of silence.

sox in.wav out7.wav silence -l 1 0.3 1% -1 2.0 1%

As a result you’ll see that we’ve cropped out the 0.25 seconds of noise at the beginning of the clip, but left the 0.5 seconds of noise in the middle.

For actual usage, you’ll probably want to specify something shorter than 0.3 seconds for the duration if you’re just trying to filter out pops and clicks.

Bonus Example 8: Splitting audio based on silence

sox in.wav out.wav silence 1 0.5 1% 1 5.0 1% : newfile : restart

Using SoX’s newfile pseudo-effect allows us to split an audio file based on periods of silence, and then calling restart starts the effects chain over from the beginning. In this example, SoX will split audio when it detects 5 or more seconds of silence. You’ll end up with output files named out001.wav, out002.wav, and so on.

Final Thoughts

There you have it.  This is what I know about the silence filter effect in SoX.  Example 7–where we trim some but not all of the silence and ignore pops and clicks–is ultimately what I was trying to figure out when writing this article, but I figure the other examples have got to be a good reference for somebody me.

The above and below-period values are still mostly a mystery to me.  I may address them in another post, but for now, I’m just going to use this as a cheat sheet in case I forget.

And don’t forget to use the trailing zero when specifying whole seconds. Even while writing this I forgot multiple times.

I welcome thoughts, ideas, comments, and corrections. Please.

(edit 11/14/10 to add names to each of the examples for clarification)
(edit 04/28/11 to add audio splitting example)
(edit 12/06/12 to add one line silence trimming) 

    • sox in.wav out.wav silence 1 0.8 1% 1 1.0 1% : newfile : restart

Categories
Software

Quietly Complaining

I’m switching from DSL to cable internet.  It’s not so much that I prefer cable internet (in fact, I like the ease of the home DSL install) but our house is in the middle of town and conveniently too far from the central offices on either side of town.  It’s only like a 2 block stretch, and I’m smack dab in the middle of it.

What this boils down to is this: the level of DSL service I’m paying for is up to 7 megs, but what I’m getting if I’m lucky is 3 megs.  My wife tried to watch a streaming Netflix movie on the Xbox 360 the other day, and every 4 minutes it stopped to rebuffer.  Lame.

So I spent the next couple of days talking to various salespeople for our local cable internet provider and I figure I might as well switch to their VoIP service rather than Vonage because I can get a bundled deal.  Might as well.  The home phone is literally only around so that telemarketers and credit card companies have another number to call rather than my cell phone.  So lets just port the number…

Porting the number is where the fail begins.

The first two salespeople I spoke with put me on hold when finding out there was some sort of problem with porting the number.  One even told me I’d just have to get a new number.  Then they offered to investigate further, so I gave them a callback number and…they never called back.

The last guy I talked to put me on hold while he escalated the issue up a couple of levels, and then told me that it didn’t appear like they could port the number even though it looked like they could on the surface.

While on hold, I came to the realization that maybe the problem isn’t with the cable internet provider and their digital phone team.  Maybe it’s a problem with Vonage.

Oh, Vonage.

So I tell the salesguy to just put an order in for the cable internet, and I’ll keep Vonage until I figure out what on earth is going on.  He’s nice enough to comp me the install fee for my troubles.

Getting off the phone, I post this to Twitter:

@Vonage_Voice number portability #fail. Apparently you’re holding my number hostage and I can’t port it to another VoIP service. Boo.

Literally not a minute later, and I get an email notification stating that Vonage_Voice is following me on Twitter, and a minute after that I see:

@tfj Can you please DM me your name and best number to reach you? Can have this looked into for you.

Impressive.

I send him my contact information, and he lets me know that someone will get a hold of me that evening or first thing tomorrow.

In the morning I get a call from Vonage’s Executive Response Team.  The woman on the phone tells me she got the ticket first thing this morning, but realized that since I’m on the west coast, I probably wouldn’t appreciate a call at 5am.  Very kind of her.

She explains the situation.  Since I didn’t port a number into the Vonage universe, but instead took a new number when I signed up, that the number I have is held with one of their resellers under the name Vonage Holdings (or something along these lines, I forget exactly).  She lets me know exactly what I need to do to port a number out from Vonage, and sends me an email with all the same info.

So I’m still with Vonage.

Ultimately, I may still switch to something else.  Google Voice looks promising but I’m waiting a few months to see how well it pans out.

But props to Vonage for their speedy resolution to my problem.  No calling support and waiting for somebody that didn’t want to tell me what I wanted to hear; no filling out a random online support form and not getting an email back.

Just a quiet little tweet that rings in the ears of the corporate giants.

Categories
Design

Autobots

More than meets the eye...
More than meets the eye...

My new Macbook sticker got linked in a MetaFilter blog post today.  Huzzah!

Categories
System Administration

Forwarding Yahoo! Mail to Gmail (For Free!)

I’ve got way too many email addresses. Free email accounts opened up as a teenager that I don’t want to get rid of because they hold some strange nostalgic value. Hotmail, Gmail, Yahoo…I’ve even got an account with RocketMail, which eventually became Yahoo! Mail.

So consolidation is my other option.  I’m using Google Apps for Domains which essentially gives me Gmail on my own domain.  Most of the free email services have some mechanism which I can use to forward to my Gmail account, but Yahoo! requires their paid service.  No fun.

If I was using a desktop email client I could just implement YPOPs! or FreePOPs. But I’m not.

Linux to the rescue! Using Exim and FetchYahoo I can now periodically check my Yahoo! Mail accounts and redirect them to my Gmail account. FetchYahoo is great because it provides the mechanism to forward emails to another email address; the products mentioned about don’t appear to have this feature.

Here’s how I did it, using Ubuntu 8.10 (Intrepid Ibex).

Reconfigure Exim4 to use Gmail SMTP (Note: POP must be enabled in your Gmail settings). I used the instructions I found on TechSutra, which I’m not going to reproduce here because I’m lazy.

Remember to comment out or remove any other smarthost defined with “domains = ! +local_domains” as well as any other authenticators where “public_name = LOGIN”. These were both defined for me and I originally missed these steps.

Ensure that Exim4 is working. See step 3 from TechSutra’s instructions.

Download and install FetchYahoo (2.13.3 as of this writing). Alternatively, install from the repos (it’s an older version though):

sudo apt-get install fetchyahoo

Test FetchYahoo. Replace <username> and <password> with your, well, username and password.

fetchyahoo --mailhost=localhost --username=<username>@yahoo.com \
--password=<password> --onlylistmessages

You should see a list of the emails in your inbox. If it bombs out, you might try logging into your Yahoo! Mail and changing the interface from Classic to New and vice-versa.

Try forwarding to Gmail using FetchYahoo. You can limit the number of emails it fetches by using –maxmessages=N where N is the maximum number of messages to grab, but I only had a handful of emails in my inbox at the time.  Once the emails are processed, they’ll end up in the Trash, so if there are problems, you can just move them back to the inbox and try again.

fetchyahoo --sendto=<username>@gmail.com --mailhost=localhost \
--username=<username>@yahoo.com --password=<password>

You should see something like the following:

Only forwarding e-mail, local delivery turned off.
Logging in securely via SSL as <username>@yahoo.com on Wed Apr 15 00:04:15 2009
Country code : us    FetchYahoo! Version: 2.13.3
Successfully logged in as <username>@yahoo.com.
Marking messages read on the server

Fetching mail from folder: Inbox
Getting Message ID(s) for message(s) 1 - 5.
Got 5 Message IDs
._._.__.5
Finished downloading 5 messages.
5 message(s) have been deleted.

You can monitor the Exim4 logfile to see if the emails are being sent properly:

tail -f /var/log/exim/mainlog

Check your Gmail account for the forwarded emails. Now the only glitch I’m having here is that forwarded emails seem to be coming into All Mail rather than Inbox and are already marked as read.  If anyone has any ideas on this, please let me know.

If all is well at this point, then congratulations! If all is not well, then I don’t know what to tell you, slick. It worked for me.

Create FetchYahoo configuration file. FetchYahoo allows for an /etc/fetchyahoorc file or ~/.fetchyahoorc file rather than passing all the options via the command line. From the FetchYahoo source folder, copy fetchyahoorc to your home directory:

cp fetchyahoorc ~/.fetchyahoorc

Then edit the file, substituting your own values as applicable.  The most important values are:

username = yahoo-user-name@yahoo.com
password = yahoo-password
use-forward = 1
mail-host = localhost
send-to = gmail-user-name@gmail.com

If you’d rather store a hashed password than the plaintext, you can run

fetchyahoo --md5hex

which will give you a md5 hash of your password suitable for storing in the configuration file.

Save and test the config file by running fetchyahoo with no parameters.

Schedule to run periodically via cron or use the daemon mode. I’m using cron because I’ll probably only check these accounts every few days and I don’t need the process just hanging around in the meantime.  The –repeat-interval=N command line parameter (or just repeat-interval = N in the config file) is all you need to run the process as a daemon.

And that’s it!  So until the Yahoo! folks decide to come around and allow for POP3 access or some other mechanism to get email out from their stronghold, this is what I’ll use to get my coveted bits of spam to Gmail.

Update: The original procedure described above ended up dropping forwarded emails from Yahoo into the All Mail folder rather than the Inbox due to the fact that I was sending to and from the same Gmail account.  Since I implemented this procedure using Google Apps for Domains, I simply created a generic account from which I am sending emails, and all is now well with the world.

Resources: