vBulletin 4.1.7 Mobile Style Giving You Grief?

I upgraded a forum to vBulletin 4.1.7 last night. Pretty straightforward stuff, only the mobile style that they half-introduced back in 4.1.2 or somewhere along the lines has been a little glitchy.  4.1.6  reintroduced the mobile style, but I blinked and 4.1.7 was released.

I ended up deleting the mobile style and reimporting it from scratch (see Installing the vBulletin Mobile Style from their manual) but the grid menu button didn’t seem to work. The grid menu is where they’ve put notifications, messages, what’s new, etc. in the mobile style. In particular, I had some users ask where the What’s New feature was in the mobile style, and that’s finally been added, so it was important to get the grid menu working.

Here’s what I found.

vBulletin Mobile Style Grid Menu Fix

Set the Default Mobile Style for Modern Browsers to your Mobile style

Under Settings -> Options -> Style & Language Settings, make sure to set the Default Mobile Style for Modern Browsers to your Mobile style. I went ahead and set as the default for Old Mobile Browsers as well, since I don’t know what the difference is.

And voila!

That is all.

Posted in Software, System Administration | Tagged , , , , , , | Leave a comment

MyFax Support, You Suck

tl;dr MyFax Support sucks. Getting ‘Mydll_NT: Error in function: dAddPortMonitor, Error code: 126.’ installing the Print-to-Fax Assistant? Run msiexec /i myfaxassistantsetup.msi /q and install in silent mode.

Dear j2 Global Communications/Protus/MyFax:

Please remind your company to care about their software and the people that use it. In particular, you need to provide support for the installation of your products. Telling someone you won’t support your own product until it’s installed is ridiculous and asinine.

What Went Wrong: Installing Print-to-Fax Assistant

A few days I attempted to install the MyFax Print-to-Fax Assistant on a fresh-out-of-the-box Dell workstation running Windows 7  Professional 32-bit SP1. Nothing out of the ordinary installed on this system. It’s joined to a domain and I’m running the install as myself, a user with administrative rights on the local machine.

I accept the EULA and all the defaults to get the install on its merry way, and I receive the following error:

MyFax Print-To-Fax Assistant Installation Error

Mydll_NT: Error in function: dAddPortMonitor, Error code: 126.

After that, the install failed and the MSI rolled back.

I tried a reboot and a few other normal troubleshooting tactics, and it all ended with the same error. I’ve got the Print-to-Fax driver installed on a Win 7 64-bit and another Win 7 32-bit install, as well as an XP box, so I know the thing works. Even tried downloading the installer again and comparison the checksum. Everything checked out, but it wouldn’t install. It’s not a particularly helpful error message, and there’s no information on any sort of manual installation process.

To the Googles! Less than 150 results come back on a search for this particular error. Mostly from Black Ice Online Support, which leads me to suspect that they provided the lower level drivers that MyFax uses. Searching for this particular error specific to MyFax returns nothing.

How Not To Help: MyFax Support

MyFax Knowledgebase

Nothing there of any relevance. In fact, it looks like the latest I can find on any subject was last modified almost a year ago in June of 2010. (Update: by the time I got around to finishing this article, I now see one new knowledgebase post from July 2011.)

Online Chat Support

So, it’s time to contact support. First, I try the online chat. Here’s where it gets interesting.

I’m told that:

  • The Print-to-Fax driver is not compatible with Windows 7 (contrary to information on the MyFax site even showing 64-bit Win 7 compatibility)
  • The Print-to-Fax driver is one of the oldest plug-ins they have
  • That Blackberry, iPhone, and Android apps are out or in the making

None of which helps in the slightest to resolve my situation.

It’s also suggested that I turn off my firewall (which it is), and then I’m given a couple of Google hits on the error, neither of which reference MyFax in any way.

Finally, I’m told that it’s not a MyFax error, and it’s the computer that’s giving me this error. Oh really? So the fact that I get this error when installing the MyFax product means nothing? How about uninstalling it and then reinstalling (except I NEVER GOT IT TO INSTALL IN THE FIRST PLACE!!!!!)? Blerg. Kthxbye.

Phone Support

Let’s see if a phone call nets anything more helpful. What follows is a highly paraphrased conversation.

I’m told by the Tier 1 tech that it’s a driver issue, and that I need to install the driver so that the Print-to-Fax assistant will work. And to try Google. Can I speak to a supervisor? Sure, he says, and puts me on hold.

Me: “Hello Mr. Supervisor? I can’t install your program!”

Supervisor: “Well, Mr. Customer, the ‘dAddPortMonitor’ error is a very common error. You need to install the Mydll_NT.dll in order to get our program to work.”

(Common? ‘dAddPortMonitor’ returns less than 300 hits on a search engine. ‘Mydll_NT.dll’ returns about 600)
Me: “I can’t download Mydll_NT.dll, because it came with your software. I can’t install your software.”

Supervisor: “Oh, you can just go to Microsoft.com or Google and search for it and download it.”

Me: “Yeah, no, I can’t.”

Supervisor: “Well we cannot support the Print-to-Fax assistant until it’s installed at which point we can check settings remotely from our systems?”

Me: “So you’re telling me you sell a product that you don’t support? If I can’t get your product installed, I can’t use your product. What about those Blackberry, iPhone, and Android apps? You’re telling me if I can’t get them installed, that’s my own problem?”

Supervisor: “Basically, yes.”

Me: “Okay, we’re getting nowhere. If you have any play at all with anyone in management, customer support, marketing, or product development, tell them that SOMETHING IS MAJORLY WRONG HERE. *click*”

Finally, a Workaround

I really shouldn’t have to do this, and MyFax support has given me nothing to go on, but I keep troubleshooting the issue. Running msiexec.exe with verbose logging options enabled, I discover that the TARGETDIR variable is ignoring the default path and is dumping all the files in the root folder. Changing the path doesn’t help, creating the path ahead of time doesn’t help, nor does copying the extracted files to the path in the middle of the install. Same error.

For kicks, I try a silent install:

msiexec /i myfaxassistantsetup.msi /q

It works. Why? No idea. Do I really care at this point? Not really. I run a test fax through, and it works. Reboot, and it still works.

What’s Still Wrong

So I got the Print-to-Fax Assistant to install. Yay. However, I don’t think I’d be writing all this if the whole support experience wasn’t so terribly, utterly, horribly, bad.

On one hand, MyFax and the whole internet faxing universe is really just an attempt to hold onto an antiquated technology that really should have been replaced by something better years ago, but amazingly, faxing still exists and Osama Bin Laden doesn’t. But that’s beside the point.

Here’s the thing: I pay for your software. Other people pay for your software too.

Granted, it’s not a lot of money every month, but it’s not like I’m mooching off free software. You’ve got a support page with numerous ways to contact you. But to tell me you won’t support me until I get the program installed…that’s just wrong. That’s like Dell telling me they can’t help me troubleshoot a brand new system that won’t boot up until I can get it turned on.

The installation process is a critical part of the distribution of software. If your program can’t be installed, then people can’t use it, and then it’s worthless. Zero stars rating. Refund.

The least you could have done was taken a note of the issue, given me a ticket number, and then ignore me. At least then I would’ve thought (initially) that somebody cared.

What Can Be Done

Start by fixing your Print-to-Fax Assistant installer.

Windows 7 has been out for two years. Make your software compatible. Simple as that. I downloaded the 64-bit version the other day, and got the same error message on my own desktop, and fixed it by running it through a silent install as above. But that’s a poor workaround. I’ve now had this error on at least four separate machines, so it can’t be just a fluke.

Next, figure yourself out as a company.

With all your buyouts, I count at least five distinct faxing services you offer. I’m not going to bother to list them here, and frankly I don’t care about your other products, because I don’t use them. If you want, consolidate all these similar products into one and then focus on making that product better.

Finally, beef up your support.

Support is your first line when customers are having problems. If they can’t resolve simple issues because they’re uninformed of the solutions, then provide training. Make it better.

Maybe you don’t care

j2 Global Communications, it’s entirely possible that MyFax doesn’t matter to you. Somehow your stocks are still are still up and you’ve apparently got money in your pockets with reported growth over a number of years, but the product I care about seems to be dying a slow and painful death. Since buying Protus in December 2010, you cut 100 employees from the Ottowa office and presumably dumped that workload on other locations. On on the MyFax and related websites, I see the following:

  • 1 new knowledgebase article in the last year
  • The last post on the MyFax Blog is from 7 months ago (January 2011) and as far as I can tell, all the actual content contributors on the blog are no longer with Protus or j2
  • A black hole of tweets between Dec 2010 and June 2011
  • Twittermonials on the main page of MyFax.com shows the last favorited tweet to be 200+ days old
  • Last released dates for the iPhone and Blackberry apps are both over a year old, and both with less than 50% ratings. There’s no Android app at all, and no acknowledgement publicly that they’re even working on an app for this platform.
  • The MyFax What’s New page shows a Fall 2010 Update, and nothing newer.

I could be wrong. Maybe you do want to make MyFax better. Maybe you do want to provide customers with a good experience in the bizarre world that merges the zombie-that-will-not-die that is faxing with this internet world that has Google+ and TouchPad fire sales.

People liked MyFax. Unsurprisingly to me, people liked MyFax better than eFax. This should mean something to you when deciding where to devote your time and attention.

If you do care, then show it.

Sincerely,

Jason Navarrete

Posted in Software | Tagged , , , , , , , | 16 Responses

0x0000007B Stop Error Installing Windows XP? Try Replacing Your CD-ROM Drive

Are you getting the dreaded Blue Screen of Death when you try to install Windows XP?

Have you already tried using a slipstreamed SP3 disk?

Does that error say something like:

STOP: 0x0000007B (0xF78D2524,0xC0000034,0x00000000,0x00000000)
Blah blah blah blah fail blah blah CHKDSK /F blah blah failblog.org

Have you already tried all the different modes of SATA/RAID/AHCI/IDE/Legacy nonsense in the BIOS to try to get your hard drive to be detected properly but the darned thing still doesn’t want to install? Already updated your BIOS to the latest and greatest?

Even tried slipstreaming RAID and/or SATA drivers on an install CD or went so far as to dig up a floppy drive to F6 some drivers in place?

Think there’s no answer?

***Try replacing your IDE CD-ROM/DVD-RW drive with a SATA drive.***

Worked for me.

It’s deceptive, I know, but apparently even though the CD starts the Windows install process and loads drivers and everything seems grand, setup takes a big dump later on. I tried all the options above, even tried an IDE hard drive and SATA raid setups in various combinations, but they all ended in a BSoD. Nothing worked until I swapped the IDE CD-ROM with a SATA CD-ROM drive.

At one when I tried to install Windows 7, I got a message stating: A required CD/DVD device driver is missing. If you have a driver floppy disk, CD, DVD or USB flash drive, please insert it now. Note. If the windows installation media is in the drive, you can safely remove it for this step.

That really should have clued me into the whole CD/DVD thing, but of course I paid no attention, because I assumed if we got that far in the installation, that it was a hard drive controller issue, not a CD-ROM issue. Ah well.

Moral of the story: Try changing your BIOS settings first. Then try changing your CD-ROM drive.

Posted in Computer Fixin' | Tagged , , , | Leave a comment

Adding Rails log rotation to Dreamhost (compiling your own logrotate)

(credit goes to Joey Geiger for the original Gist)

Trying to use logrotate to rotate your Rails logs on Dreamhost?   You’ll need to roll your own copy of it.  Fortunately, the process isn’t too bad.  Let’s jump right into it.

SSH to your host and run the following commands (read my notes below if you’re not super gusty):

# create a new cron job (crontab -e) or use panel.dreamhost.com and run it daily
0 0 * * * /home/<your_account>/opt/bin/logrotate -f -s /home/<your_account>/opt/lib/logrotate.status /home/<your_account>/opt/etc/logrotate.conf
cd ~
mkdir -p opt/bin
mkdir opt/lib
mkdir opt/etc
mkdir src
cd src
wget http://www.sfr-fresh.com/unix/privat/logrotate-3.7.9.tar.gz
wget http://www.sfr-fresh.com/linux/misc/popt-1.16.tar.gz
tar xzvf logrotate-3.7.9.tar.gz
tar xzvf popt-1.16.tar.gz
cd popt-1.16
./configure --prefix=/home/<your_account>/opt/
make
make install
cd ../logrotate-3.7.9
# the first make will fail on the final compile (with 'cannot find -lpopt' message)
# some include problem, but this works so i'm not inclined to care too much
make POPT_DIR=/home/<your_account>/opt/include
# this make should complete
make POPT_DIR=/home/<your_account>/opt/lib
cp logrotate ~/opt/bin/
cd ~/opt/etc
vi logrotate.conf
# paste in the logrotate.conf file
/home/<your_account>/<your_website>/shared/log/*.log {
  daily
  missingok
  rotate 7
  compress
  delaycompress
  notifempty
  copytruncate
}


A couple of quick notes here:

    The most substantial difference from jgeiger’s post is the compiling of the popt library, so that in turn, logrotate would compile.  I really don’t know what’s messed up when you try to link to popt (which leads to the compile failing the first time around), but the workaround seems to work just fine.  If someone happens to know what magic flags are necessary to get the compile to work smoothly the first time, please fork the notes and let me know.
    Joe’s Own Editor was the default editor when I tried to edit my crontab, and it makes my brain hurt.  Edit your .bash_profile and add export EDITOR="/usr/bin/vim" to use vim instead. Logout and back in for the change to take effect.

(disclaimer: Don’t break your stuff. Or Dreamhost’s stuff.)

Posted in System Administration | Tagged , , , | Leave a comment

Nagios and NConf on Ubuntu 10.04 Lucid Lynx

As of December 2011, NConf has released version 1.3.0 which changes the deployment methods for configs. Take a look at my NConf 1.3.0 upgrade guide!

Have you been wondering about how to get Nagios and NConf setup on your Ubuntu server?  Well, I took some notes the last time around, and here they are, for anyone that may find them useful.  Here we go!

(disclaimer: don’t screw up your system.)

Prerequisites

If you didn’t check the box to configure your box as a LAMP (Linux, Apache, MySQL, PHP) server when you first setup your box, do that first with:

sudo apt-get install apache2
sudo apt-get install mysql-server mysql-client
sudo apt-get install php5 libapache2-mod-php5

If you want phpMyAdmin (for web-based MySQL administration):

sudo apt-get install phpmyadmin

Installing Nagios

Install Nagios with

sudo apt-get install nagios3

It should install all the necessary dependencies. Follow the prompts to get it configured and test it out by browsing to http://<yourserver>/nagios3

If you would like to monitor Windows servers using NSClient++ (which I am doing), also install the NRPE plugin with

sudo apt-get install nagios-nrpe-plugin

I felt like I ought to backup the config files before I started messing with them, so I backed them up with

sudo cp -r /etc/nagios3 /etc/nagios3.backup

Installing NConf

NConf provides a web-based frontend for configuring Nagios.  There’s no package in the Ubuntu repositories for it, but you can download it pretty easily.  Version 1.2.6 was the latest as of the time of this writing, so make sure you’re getting the latest version.  (Note: it looks like the link below gets cut off, so here’s a link to the NConf download page where you can grab the latest source)

wget http://sourceforge.net/projects/nconf/files/nconf/1.2.6-0/nconf-1.2.6-0.tgz/download
Unpack it to your webroot:
sudo tar xzvf nconf-1.2.6-0.tgz -C /var/www
Change the owner of the folder and files you just extracted to the web user:
sudo chown -R www-data:www-data /var/www/nconf
Now you’ll need to create a MySQL database to hold the NConf configuration data.  Using phpMyAdmin (sudo apt-get install phpmyadmin if you don’t have it) do the following:
  • Privileges -> Add a new User
  • User name: nconf
  • Host: localhost
  • Password: (generate, and make a note of the generated password)
  • Create database with the same name and grant all privileges
  • Go
  • Privileges -> reload the privileges

Then browse to http://<yourserver>/nconf and follow the prompts to finish initial configuration of NConf.

When you get to the database configuration page, enter nconf for the username and the database name, and use the generated password you should have made a note of earlier. For most everything else, accept the defaults, but change the NAGIOS_BIN variable to /usr/sbin/nagios3 to reference the right location.

Finally, remove the folders and files referenced at the end of the installation process:

sudo rm -r /var/www/nconf/INSTALL
sudo rm /var/www/nconf/INSTALL.php
sudo rm -r /var/www/nconf/UPDATE
sudo rm /var/www/nconf/UPDATE.php

At this point, you should be able to login to NConf, although it won’t be doing anything of importance.

Configuring Nagios to use NConf

We’ll need to make some changes to one of the Nagios configuration files, so (using sudo) open up /etc/nagios3/nagios.cfg in your favorite editor and delete or comment out all the lines that begin with cfg_dir= or cfg_file= and add the following lines:

cfg_dir=/etc/nagios3/global
cfg_dir=/etc/nagios3/Default_collector

Back at the terminal, run the following command to create a folder for NConf to dump the configuration files it generates.

sudo mkdir /etc/nagios3/import

Configuring NConf to Deploy Nagios Configurations Automatically

Almost there. Using sudo, open up /var/www/nconf/ADD-ONS/deploy_local.sh and make the following changes to paths:

OUTPUT_DIR="/var/www/nconf/output/"
NAGIOS_DIR="/etc/nagios3/"
...
/etc/init.d/nagios3 reload

GM and Allison pointed out in the comments that they had to set the execute bit on this file:

chmod +x /var/www/nconf/ADD-ONS/deploy_local.sh

This script will deploy the generated configuration package and then reload the running instance of Nagios, but it’s easiest to use just installed in the root crontab.

sudo crontab -e

and adding the line

* * * * * /var/www/nconf/ADD-ONS/deploy_local.sh

Final Steps

After saving and closing the root crontab, log back into NConf and take a look around.  You’ll see some sample definitions and some predefined services for the localhost computer.  You may want to delete the check_local_mrtgtraf and check_local_procs services, as the first one doesn’t work without additional configuration and the second one is a sample definition, but you can make those changes at your leisure.

Once you’re ready, click Generate Nagios config, and if all goes well, you’ll see something like the following:

[ Initializing NConf perl-API (library version 0.2, written by A. Gargiulo) ]
[ Copyright (c) 2006-2009 Sunrise Communications AG, Zurich, Switzerland    ]

[INFO]  Starting generate_config script
[INFO]  Generating global config files
[INFO]  Generating config for Nagios-collector 'Default Nagios'
[INFO]  Ended generate_config script

Running syntax check:

	  Default_collector:	 Total Warnings: 0  Total Errors: 0
Changes updated successfully.

Now log back into Nagios and click on Service Detail.  Within a minute or two, you should see the hosts and services change to reference the configuration as generated from NConf.

If that works, then huzzah!

Start making your configuration changes in NConf and enjoy not having the manipulate those Nagios conf files by hand anymore!

Troubleshooting, Tips and Caveats

Try running the deploy_local.sh script by hand (with sudo) if it doesn’t appear that Nagios is getting the configurations from NConf. You may be able to glean some information from the output of that script.

New hosts not showing up in Nagios after being created in NConf? Make sure you’ve selected Default Nagios under “monitored by” when defining the host, or that host won’t get assigned properly.

This setup process effectively disables all the command definitions provided by the package install of Nagios, which are stored under /etc/nagios-plugins/config.  I looked through them and compared them to the ones provided by default with NConf, and I was fine with what NConf provided.  NConf provides a mechanism to import command definitions if you really find that you need them.

I have no idea how this setup will hold up under an upgrade of the Nagios package.  We’ll see when the time comes, and if I remember, I’ll update these notes.

If you have any trouble with the steps provided above, please comment and I’ll do what I can to assist.

More Information

Take a look at the following documentation from NConf:

Posted in System Administration | Tagged , , , , | 67 Responses

Typography and the Ag Museum

So I got a new lens for my Nikon (a 50mm f/1.8D AF) and boy is it a lot of fun.  The D60 body doesn’t have an in-body autofocus motor, which is probably one of reasons why it’s so small, but it’s been a pleasant challenge forcing myself to manually focus with this lens.

These are a few of the resulting shots, taken at my local agricultural museum.

Super Chief

Armall

Regular

Bombardier Limitee

View the full set on flickr.

Posted in Photography | Tagged , , , , | Leave a comment

Star Wars Mafia

Mafia (or Werewolf or Assassin) is one of those party games that almost inevitably arises when there is a good group of more than five people (probably tending towards the geeky) get together with not a whole lot to do.  It’s a socially engaging game that can easily burn up an hour or two of an evening.

If you’ve never played, check out the Wikipedia article on Mafia to get some background information, or this Wikihow article on how to play.

The following is one of the many variants of Mafia, which happens to use everybody’s favorite Star Wars characters instead of the mafia or werewolves.  It was introduced to me by Peter and David Lindstrom, and the information here is more or less verbatim as I got it from them.  I’m posting it as a reference to myself and anyone else who might be interested to play this version of Mafia.

If anyone has any questions about the Star Wars Mafia variant, please post them in the comments and if I can’t provide insight, I’ll get a hold of Peter and David and find out from them. Enjoy!

The Force is the equivalent of the Narrator or Moderator in the standard Mafia game. Depending on the size of the group, the last few characters on the Rebel Alliance side may not be used.

The Rebel Alliance

  • Jedi Knight—Once per night the Jedi knight may either question or challenge. To question, point to a person. 10 challenge, make a sign like you are fighting with a light saber, and then point to a person. If the Jedi Knight elects to question, the Force will tell him whether the person he indicated is good or bad. A thumbs up means good. Thumbs down means bad. If the Jedi Knight elects to challenge a character, he will win if the character is bad. The bad character will be dead the next morning. However, if the Jedi Knight challenges a good character, he will die.
  • Obi Wan Kenobi—Can talk after he is dead.
  • Chewbacca—Once per game Chewbacca may interrupt discussions or a trial and eat anyone he chooses. If he tries to eat Han, Han may reveal his card and say “Will somebody get this big walking carpet out of my way?” Chewy can then choose to eat him or not. Night falls immediately thereafter.
  • Thermal Detonator—If the thermal detonator is killed during the day, he kills the person to his left. If he is killed at night. he kills the first Sith to his left.
  • Lando Calrisian—Lando may elect to smuggle once per game, during the night. If he elects to smuggle, the Force will exchange the identities of two people during the night. Only those people whose identities have changed will be aware of the switch.
  • Princess Leia—If Princess Leia is on trial, she may reveal her card, and say “Will somebody get this big walking carpet out of my way?” Han Must then reveal his card and save her.
  • Han Solo—Must save Princess Leia if she asks. If Chewy tries to eat him, he may reveal his card and say “Chewy, you stupid walking carpet.”
  • Bounty Hunter (I know, he’s not really a Rebel, but for the sake of the game he’s on this side)—Picks one person during each night to be his death buddy. If the bounty hunter dies during the night, his death buddy also dies.
  • Yoda—Raises somebody from the dead once per night.
  • Owen & Beru Lars—Die the first night.
  • Salacious Crumb—Must vote as Jabba votes.
  • Jabba the Hut—Controls Salacious Crumb’s vote
  • Jar Jar Binks—Once his identity is revealed by the probe droid, he must speak in a fake Jamaican accent. Once per night he points to someone else. Then they also must speak with an accent.

The Dark Side

  • Darth Vader—Conspires with the Emperor to kill one person each night.
  • The Emperor—Conspires with Darth Vader to kill one person each night.
  • Apprentice—Takes over for the Emperor if the Emperor dies.
  • Probe Droid—Reveals one identity per night. Everyone knows what character has been revealed, but only the probe droid knows which person has that identity.
  • Storm Trooper—Can never die. He just keeps reappearing. He doesn’t need to be killed for the good side to win.

First Night

  • Salacious Crumb and Jabba see each other.
  • Darth Vader and the Emperor see each other.
  • Darth Vader, the Emperor, and the Probe Droid see each other.
  • The Apprentice and the Emperor see each other.

Every Night

  • Yoda may bring someone back to life.
  • Lando may smuggle (if he has not already done so)
  • Jedi Knight may question or challenge
  • Bounty Hunter picks his death buddy
  • If Jar Jar has been revealed, he picks somebody to speak fake Jamaican
  • The Sith kill somebody
  • Probe Droid reveals an identity
Posted in Uncategorized | Tagged , , , | 1 Response

Quick and Dirty (Free)DOS Bootable USB Drive Under Linux

I found myself needing to flash a BIOS today, which required a DOS or Windows 98 bootable floppy disk.  Forget that.  Let’s use a USB drive instead!

Googling around returned methods that seemed way too involved for what I was trying to do.  Here’s the three commands I ended up using:

wget http://www.fdos.org/bootdisks/autogen/FDOEM.144.gz
gunzip FDOEM.144.gz
sudo dd if=FDOEM.144 of=/dev/sdb

Be sure to change /dev/sdb to whatever your USB drive shows up as. (Disclaimer: I’m not responsible if you end up nuking your secondary hard drive)

Then unplug the drive and plug it back in.  Unzip your files onto the drive (mine shows up as /media/FREEDOS), eject, and flash away!

You’ll end up with about 1.3 MB of free space on the drive, which should be enough for the majority of things.

Posted in Software | Tagged , , , , | 1 Response

Linux ‘rename’ utility on OS X

(edit 12/11/2012 to add homebrew alternative)

So you you want the powerful Perl-based rename utility that comes with many Linux and *BSD distributions, but sadly not on Apple’s mighty OS X? Well so did I. (Go away you fanboys of all types. I hear not your rants about which operating system is for the win)

Thinking about it for a moment, I realize there’s nothing really stopping me from just taking the script, and thankfully it’s free software and we can pretty much do whatever we need to with it.

The rename utility is tucked away under /usr/bin/ on my Ubuntu machine.  To get it to your OS X machine, it’s just a matter of copying /usr/bin/rename on a Linux box and dropping it in /usr/local/bin/ or somewhere else that’s in your PATH on your OS X box.  If you’re really feeling the need for the man page, you can grab it from /usr/share/man/man1/rename.1.gz.  I stuck the man page in /usr/local/share/man/man1/.

And that’s it! You can now go about using Perl regular expressions to rename multiple files and get lost in everything that is regexp.

Don’t have access to a Linux machine, you say? Oh fine. Here’s what I pulled from my Ubuntu 9.04 distro:

If you download the rename utility from here, you’ll probably need to run ‘chmod a+x’ to give execute permissions to the file after copying it to the appropriate place but otherwise things should be the same as above.

Examples

Wondering how it works?  Say you’d like to rename a bunch of .wav files and prepend the phrase ‘iheartrobots’ to the front of each filename (practical, I know).  Well you’d type the following:

rename -v 's/^/iheartrobots/' *.wav

The ‘-v’ parameter prints the names of the files that are being renamed.  Helpful and I use it most of the time.

Or how about replacing all instances of the word ‘cat’ with the word ‘dog’ in the filename of all .rtf files in a folder? Bam.

rename -v 's/cat/dog/i' *.wav

The ‘i’ at the end of the regular expression makes the search case-insensitive.

Finally, say you had a bunch of files that looked like this: Waltzing.With.Famous.People.S01E09.HDTV.XviD-SYS.[bztv].avi

And you wanted to get rid of everything between S01E09 and .avi, you could use the following:

rename -v 's/HDTV.*avi/avi/i' *.avi

The Homebrew Alternative

Now, thanks to Homebrew, you can skip all this nonsense and just do:

brew install rename

If you’re not using Homebrew, you should be.

More Information

If you’d like to get more information on the rename utility described here, the most obvious place is the man page (man rename) or online here.

Additionally, since it’s a Perl utility, more information on the Perl regular expression pattern-matching language can be found straight from the horse’s mouth.  Regular expressions are a madness unto themselves, but once you get the hang of it, most powerful stuff.

Posted in Software | Tagged , , | 2 Responses

The SoX of Silence

SoX is, by their own definition, the Swiss Army knife of audio manipulation.

And no doubt it’s full of fun with slicing and dicing and playback and recording and filtering and effects capabilities.

But SoX is a command line tool, which means obscure syntax and parameters in order to get things done.

I’ve been trying off and on for months to try to understand the silence filter from within SoX, which allows one to remove silence from the beginning, middle, or end of the audio. Sounds, simple, doesn’t it?  Well, it should be.

Below is the man page for the silence filter:

silence [-l] above-periods [duration threshold[d|%] [below-periods duration threshold[d|%]]

Removes silence from the beginning, middle, or end of the audio. Silence is anything below a specified threshold.

The above-periods value is used to indicate if audio should be trimmed at the beginning of the audio. A value of zero indicates no silence should be trimmed from the beginning. When specifying an non-zero above-periods, it trims audio up until it finds non-silence. Normally, when trimming silence from beginning of audio the above-periods will be 1 but it can be increased to higher values to trim all audio up to a specific count of non-silence periods. For example, if you had an audio file with two songs that each contained 2 seconds of silence before the song, you could specify an above-period of 2 to strip out both silence periods and the first song.

When above-periods is non-zero, you must also specify a duration and threshold. Duration indications the amount of time that non-silence must be detected before it stops trimming audio. By increasing the duration, burst of noise can be treated as silence and trimmed off.

Threshold is used to indicate what sample value you should treat as silence. For digital audio, a value of 0 may be fine but for audio recorded from analog, you may wish to increase the value to account for background noise.

When optionally trimming silence from the end of the audio, you specify a below-periods count. In this case, below-period means to remove all audio after silence is detected. Normally, this will be a value 1 of but it can be increased to skip over periods of silence that are wanted. For example, if you have a song with 2 seconds of silence in the middle and 2 second at the end, you could set below-period to a value of 2 to skip over the silence in the middle of the audio.

For below-periods, duration specifies a period of silence that must exist before audio is not copied any more. By specifying a higher duration, silence that is wanted can be left in the audio. For example, if you have a song with an expected 1 second of silence in the middle and 2 seconds of silence at the end, a duration of 2 seconds could be used to skip over the middle silence.

Unfortunately, you must know the length of the silence at the end of your audio file to trim off silence reliably. A work around is to use the silence effect in combination with the reverse effect. By first reversing the audio, you can use the above-periods to reliably trim all audio from what looks like the front of the file. Then reverse the file again to get back to normal.

To remove silence from the middle of a file, specify a below-periods that is negative. This value is then treated as a positive value and is also used to indicate the effect should restart processing as specified by the above-periods, making it suitable for removing periods of silence in the middle of the audio.

The option -l indicates that below-periods duration length of audio should be left intact at the beginning of each period of silence. For example, if you want to remove long pauses between words but do not want to remove the pauses completely.

The period counts are in units of samples. Duration counts may be in the format of hh:mm:ss.frac, or the exact count of samples. Threshold numbers may be suffixed with d to indicate the value is in decibels, or % to indicate a percentage of maximum value of the sample value (0% specifies pure digital silence).

The following example shows how this effect can be used to start a recording that does not contain the delay at the start which usually occurs between `pressing the record button’ and the start of the performance:

rec parameters filename other-effects silence 1 5 2%

Huh?

So lets try to clarify some of the mess from the man page.  First a couple of important notes:

  • When specifying duration, use a trailing zero for whole numbers of seconds (ie, 1.0 instead of 1 to specify 1 second). If you don’t, SoX assumes you’re specifying a number of samples.  Who on earth would want to specify samples instead seconds? You got me. Alternatively, you can specify durations of time in the format hh:mm:ss.frac.
  • Use at 0.1% at a minimum for an audio threshold. Even though 0% is supposed to be pure digital silence, with my test file I couldn’t get silence to trim unless I used a threshold larger than 0%. If you’d like, you can specify the threshold in decibels using d (such as -96d or -55d).
  • The realistic values for the above-period parameter are 0 and 1 and values for the below-period parameter are pretty much just -1 and 1. The documentation states that values larger than 1 can be used, but it only really makes sense for files with consistent audio breaks. Just trust me, it’s weird. I’ll get into what those values actually mean in the examples.

Now onto some examples! I’ll be showing you visually what happens to a sound file when we apply the various parameters to the silence filter.

I generated a test sound file with 60 seconds of white noise and then silenced various parts of the clip, leaving me with an audio file that looks like this:

SoX Silence Example (Original File)

Example 1: Trimming silence at the beginning

sox in.wav out1.wav silence 1 0.1 1%

The above-period parameter is first after the silence parameter, and for the sake of this article, it should be set to 1 if you want to use the filter. This example roughly translates to: trim silence (anything less than 1% volume) until we encounter sound lasting more than 0.1 seconds in duration. The output of this command produces the following:

sox in.wav out1.wav silence 1 0.1 1%

We’ve lopped off the silence at the beginning of the clip. For simplicity’s sake, we’ll refer to the 1% threshold as silence from now on.

Example 2: Ignoring noise bursts

sox in.wav out2.wav silence 1 0.3 1%

By changing the duration parameter to 0.3, we tell SoX to ignore the burst of noise at the beginning of the example clip. This produces the following:

sox in.wav out2.wav silence 1 0.3 1%

We can ignore short pops and clicks in audio by adjusting this duration parameter.

Example 3: Stopping recording when no sound detected

sox in.wav out3.wav silence 1 0.3 1% 1 0.3 1%

Now we introduce the below-period parameter it’s respective sub-parameters.  Just like the above-period parameter, just set it to 1 and call it good.  The command above translates to: trim silence until we detect at least 0.3 seconds of noise, and then trim everything after we detect at least 0.3 seconds of silence.

sox in.wav out3.wav silence 1 0.3 1% 1 0.3 1%

This returns a file with just the first 4 seconds of noise (note that we ignore that 0.25 sec burst of noise at the beginning). Where’s the rest of the clip?  Well, it’s gone. Not super practical for post-production of audio, but can be useful when recording live audio, so that SoX stops when it doesn’t encounter sound for a certain number of seconds.

So an aside: if you’re looking to trim silence from the beginning and the end of a audio file, you’ll need to utilize the reverse filter and a temp file like so:

sox in.wav temp.wav silence 1 0.1 1% reverse
sox temp.wav out.wav silence 1 0.1 1% reverse

Don’t forget to delete that temp.wav file when you’re done.

Jakob points out in the comments that you can trim silence from both ends in one fell swoop by chaining the effects like so:

sox in.wav out.wav silence 1 0.1 1% reverse silence 1 0.1 1% reverse

Example 4: Trimming all silence

sox in.wav out4.wav silence 1 0.1 1% -1 0.1 1%

By changing the below-period parameter to -1, we can trim instances of silence in the middle of the clip, by allowing the filter to restart after it detects noise of the specified duration.

sox in.wav out4.wav silence 1 0.1 1% -1 0.1 1%

In my example clip, it’s impossible to detect where the silence used to be, but with an actual podcast or other audio, it should be easier to tell.

Example 5: Ignoring short periods of silence

sox in.wav out5.wav silence 1 0.1 1% -1 0.5 1%

In similar fashion as Example 2, we can instruct SoX to ignore small moments of silence (1/2 second in this example).

sox in.wav out5.wav silence 1 0.1 1% -1 0.5 1%

When trimming silence from podcasts and the like, this prevents you from removing moments when someone stops to take a breath and making the conversation sound too rushed.

Example 6: Shortening long periods of silence

sox in.wav out6.wav silence -l 1 0.1 1% -1 2.0 1%

So what if you wanted to just shorten long moments of silence rather than remove them entirely?  Well, you need to add the -l parameter, but it needs to be placed first, before the other parameters for the filter effect. The example above results in trimming all silence longer than 2 seconds down to only 2 seconds long.

sox in.wav out6.wav silence -l 1 0.1 1% -1 2.0 1%

Note that SoX does nothing to bits of silence shorter than 2 seconds.

Example 7: Shortening long periods of silence and ignoring noise bursts

sox in.wav out7.wav silence -l 1 0.3 1% -1 2.0 1%

Finally, let’s tie it all together by trimming silence longer than 2 seconds down to 2 seconds long, but ignore noise such as pops and clicks amidst the moments of silence.

sox in.wav out7.wav silence -l 1 0.3 1% -1 2.0 1%

As a result you’ll see that we’ve cropped out the 0.25 seconds of noise at the beginning of the clip, but left the 0.5 seconds of noise in the middle.

For actual usage, you’ll probably want to specify something shorter than 0.3 seconds for the duration if you’re just trying to filter out pops and clicks.

Bonus Example 8: Splitting audio based on silence

sox in.wav out.wav silence 1 0.5 1% 1 5.0 1% : newfile : restart

Using SoX’s newfile pseudo-effect allows us to split an audio file based on periods of silence, and then calling restart starts the effects chain over from the beginning. In this example, SoX will split audio when it detects 5 or more seconds of silence. You’ll end up with output files named out001.wav, out002.wav, and so on.

Final Thoughts

There you have it.  This is what I know about the silence filter effect in SoX.  Example 7–where we trim some but not all of the silence and ignore pops and clicks–is ultimately what I was trying to figure out when writing this article, but I figure the other examples have got to be a good reference for somebody me.

The above and below-period values are still mostly a mystery to me.  I may address them in another post, but for now, I’m just going to use this as a cheat sheet in case I forget.

And don’t forget to use the trailing zero when specifying whole seconds. Even while writing this I forgot multiple times.

I welcome thoughts, ideas, comments, and corrections. Please.

(edit 11/14/10 to add names to each of the examples for clarification)
(edit 04/28/11 to add audio splitting example)
(edit 12/06/12 to add one line silence trimming) 

    • sox in.wav out.wav silence 1 0.8 1% 1 1.0 1% : newfile : restart

Posted in Software | Tagged , , | 103 Responses