Scrolling Horizontally

If you would like to scroll horizontally with the cursor following the screen, you can use the following commands to scroll left and right.

zl - move the view [count] characters right
zh - move the view [count] characters left

Count Words

If you would like to count the number of words in the current Vim buffer, you can simply issue the following command: g CTRL-G (no space between g and CTRL-G).

Nautilus MP3 Column Display

There have been a few instances when I've felt it would be useful to have Nautilus, Gnome's file browser, display MP3 metadata info. It's particularly nice when organizing large volumes of files by directory. KDE has had this functionality for a while, but it's been suspiciously absent from Gnome. I found a quick hack, which allows you to add metadata columns at will.

Nautilus MP3 Column Display

Daily Vim Song of the Week

I can't get enough of this song.

The Reigning Sound - Find Me Now

If anybody else has a song of the week, post it here ;-).

Uppercase, Lowercse, Swapcase

In Vim, you can use the following motions to modify the case of an entity.

gum - lowercase text operated on by motion
gUm - uppercase text operated on by motion
g~m - switch case operated on by motion
~   - if notildeop is set (default), switch case under cursor
~   - if tildeop is set, switch case of {motion} text

Examples:

guw - lowercase current word
gU$ - uppercase to end of line
g~w - swapcase current word
v$~ - swapcase to end of line (if notildeop is set)
~w  - swapcase of word (if tiildeop is set)

mk-table-checksum and Columns

UPDATE: I submitted a patch to the Maatkit developers, and the column bug mentioned below if fixed in SVN.

At work, we've been struggling with MySQL replication losing synchronization for the last few months. It's usually something like a few timestamps being off here and there, but we've also seen primary keys with artificially high values on the slave, certain replication events just not happening, and other weirdness. One toolset that has been very helpful in figuring out when problems occur is Maatkit. It a suite of tools written in Perl, which allow you to do things like run checksums on the master and slave for a given table, compare, and see if they're out of sync.

Unfortunately, some of the documentation is very sparse, which leads me to this blog entry. Google had nothing useful regarding how to use the columns feature, which allows the checksum to be calculated only on specific columns. I post this here in hopes it will help somebody.

We had been trying to use mk-table-checksum in the following way:

mk-table-checksum 127.0.0.1 -usomeuser -p'password' -tmydb.TableOne --replicate=maint.checksum --columns=UserID,TSAdded

Unforunately, running it this way would give a warning:

Use of uninitialized value in concatenation (.) or string at /usr/bin/mk-table-checksum line 431.

and produce no results. I finally got annoyed enough that I fired up the Perl debugger and stepped through what was going on. As it turns out, the data stored internally in $table->{cols} is all lowercase. If you pass in your column list camel-capped, or uppercase, it simply won't work.

The following invocation would produce useful results:

mk-table-checksum 127.0.0.1 -usomeuser -p'password' -tmydb.TableOne --replicate=maint.checksum --columns=userid,tsadded

Stupidly simple, but not necessarily obvious.

Bash, Re-writing a Line

There are instances when writing a shell script that you want to write to the same line over and over. This is mainly used for progress meters that update the same screen position over and over. Most languages represent a carriage return without newline as the \r escape sequence. In shell, it's no different, but there is one small caveat. The echo command requires two arguments.

#!/bin/bash

# simplest progress meter
items=100000
for ((i=0; i<$items; i+=1)); do
    echo -n -e "Processed $i/$items\r"
done

The -n argument to echo tells it to suppress the implicit newline that it usually includes. The -e argument tells it to respect escape sequences such as \r.

Perl Broken on CentOS 5.2 and Fedora 9

I'm not really a fan of re-blogging stuff, but I found this so staggeringly stupefying that I felt the need to post it here.

Redhat Perl, What A Tragedy

We run a variety of Perl scripts on our production systems here. A quick test using the script from the article shows that a basic loop calling bless 50,000 times takes 0.16 seconds on my Ubuntu workstation. Meanwhile, our uber-powerful production server, which runs a Redhat derivative, takes over four seconds to run the script. Our development server, running the same OS on slightly less powerful hardware took 24 seconds.

To summarize, if you run CentOS or Fedora, and you use Perl in production, you may have a problem.

Sharing a Session

You can use script and mkfifo to allow a second session to supervise the first using this very simple hack.

# on 1st terminal
mkfifo /tmp/session
script -f /tmp/session

# on 2nd terminal
cat /tmp/session

Now the 2nd terminal can interactively watch whatever is going on with the 1st terminal. Occasionally some control characters will get a little garbled when a lot of curses activity is going on, but in general, it works pretty well.

Installing Ruby DBI, MySQL on Centos 5.2

There's no good documentation on how to do this online, so here's a quick tutorial on how to get Ruby's DBI and MySQL packages installed on CentOS 5.2. There aren't any pre-packaged binaries for this, so building from scratch is pretty much your only option.

# download
wget http://tmtm.org/downloads/mysql/ruby/mysql-ruby-2.7.tar.gz

# extract
tar zxvf mysql-ruby-2.7.tar.gz

# change dir
cd mysql-ruby-2.7

# configure (requires mysql development package AND gcc)
ruby extconf.rb --with-mysql-config
checking for mysql_ssl_set()... yes
checking for mysql.h... yes
creating Makefile

# finish (as root)
make install
(ignore warnings)

Now DBI:

# download
wget http://rubyforge.org/frs/download.php/41303/dbi-0.2.2.tar.gz

# extract
tar zxvf dbi-0.2.2.tar.gz

# change dir
cd dbi-0.2.2

# setup
ruby setup.rb config --with=dbi,dbd_mysql
ruby setup.rb setup

# finish (as root)
ruby setup.rb install

Ubuntu + Bootlogd = ?

Apparently the last three releases of Ubuntu have been plagued by issues with bootlogd. In fact, it's been such an issue that boot logging is now disabled by default. I like to see what's going on under the hood during bootup, so I Google'd around and compiled the following tips to get it working again.

1) This thread contained a custom tarball of an older version of bootlogd. Grab the tarball from here. Then run the following commands (assuming it's saved in /tmp).

# unpack the source
tar xzvf bootlogd_2.86.02.tar.gz
# build the package
cd bootlogd-2.86.02
debuild -us -uc -b
cd ..
# install the package
sudo dpkg -i bootlogd_2.86.02_*.deb

2) After that, make sure it's setup at the appropriate priority in the init system.

update-rc.d -f bootlogd remove
update-rc.d bootlogd defaults 08

3) Finally, make sure it's enabled.

echo "BOOTLOGD_ENABLE=Yes" > /etc/default/bootlogd

The custom version of bootlogd writes it's log to /var/log/bootmsg instead of /var/log/boot. One final gotcha: initially, I tried running bootlogd as root from inside of X, and I got the following message:

bootlogd: cannot find console device 136:1 in /dev

This is because it's not being run from the console and does not mean that it's broken. Do a ctrl-alt-F1, login as root, and you can test running it from there.

What's Stealing That Port?

If something is bound to a port, and you want to know what it is, you can use the fuser command to determine what's attached. This is particularly handy if some rogue process has stolen port 80, and you want to know what it is.

root@travis-ubuntu:/home/travis# fuser -v http/tcp
USER PID ACCESS COMMAND
http/tcp: root 5950 F.... apache2
www-data 19140 F.... apache2
www-data 21501 F.... apache2
www-data 21508 F.... apache2
www-data 21521 F.... apache2
www-data 23207 F.... apache2
www-data 23209 F.... apache2
www-data 23245 F.... apache2
www-data 23353 F.... apache2
www-data 23460 F.... apache2
www-data 23555 F.... apache2

Reverse A File

Sometimes it's useful to cat a file in reverse order. An example would be doing things like parsing log files and such. I had a need to do this today and stumbled onto the cleverly named tac program. Usage is simple:

tac somefile.txt > somefile_reversed.txt

Preserving History

If you use more than one terminal during your workday, you've probably been annoyed that your shell history always seems incomplete. This is due to most shells overwriting existing history data in lieu of their own history when a session is closed. Fortunately, both bash and zsh support variables to avoid this behavior.

For bash, you should be able to just add the following to your bashrc:

shopt -s histappend
PROMPT_COMMAND='history -a'

For zsh, I use the following history related options:

setopt appendhistory histignoredups
setopt histignorespace extended_history
setopt inc_append_history share_history

Interactive Linux Kernel Map

Thanks to Nate for sending in this awesome map of the Linux kernel. To make it even more fun, you can scroll your mouse wheel to zoom in and out of the different components of the system.

Interactive Kernel Map

Autoplay

After a lot of hardwork and research, Grooveshark, the company I work for has released the next iteration of Grooveshark Lite, our web-based music discovery service. The newest release of the service supports an Autoplay feature, which generates recommendations based upon your listening habits. At the simplest level, you can just drop a few songs in your queue, click Autoplay, and the system will provide a continuous stream of songs based on feedback you provide.

The system uses a combination of statistically generated relational data and expert driven recommendations to provide what we hope is a very enjoyable play experience. Of course, this is a beta, so forgive me if it's not always 100% to your liking. Myself and a few other employees have had a large role in the design of this system, and I have to say that it's been one of the more interesting and challenging projects I've worked on.

Relevant Links:

Blorge

Grooveshark Lite, Featuring Autoplay

Mashable Review

Download Squad Review

Profy Review

Fayer Wayer

Seans Rant

Some Interesting Artists I've found using Autoplay:

Kathleen Edwards

The Bird & The Bee

The Shylights

The National

Gogol Bordello

Making K Useful

The default behavior of the K command is to lookup the keyword under the cursor using man. This is handy at times, but it's not very useful when I spend 90% of my day writing PHP code. Vim's online help mentions that you can set keywordprg to an alternative program, so I did the following.

set keywordprg=/home/travis/pdoc

Where pdoc is the following shell script:

 #!/bin/sh
firefox http://us3.php.net/manual/en/function.$1.php

Now I can move the cursor over a builtin function, hit K, and firefox will open to the appropriate documentation. To keep things strictly textual, you could use lynx (or links) in lieu of firefox.

For ruby code, I like to do the following:

set keywordprg=ri

For Perl:

set keywordprg=perldoc\ -tf

Recovering a Deleted Ext3 File

A co-worker accidentally deleted a .java file containing several hours worth of work. The file resided in his home directory, which conveniently had it's own dedicated partition. We managed to recover the file using the following (hackish) technique.

1) Take the machine to single user mode.

telinit 1

2) Unmount the partition containing the data.

umount /dev/sda2

3) Identify a semi-unique pattern located in the file.

His code contained a string that matched "Dictionary()".

4) Determine how much context around the string you need.

We chose 250 lines before and after the match.

5) Run grep on the partition.

grep --binary-files=text -250 "Dictionary()" < /dev/sda2 > /tmp/dump.bin

If you're lucky, the data will be somewhere in the file. We extracted the Java code using Vim, wrote a new file to /root, and went back to runlevel 2. One warning, remember not to write the recovered file to /tmp as it's often cleared when changing runlevels and rebooting.

GOTO Column

You can quickly jump to a specified column in Vim by simply typing the column number followed by a pipe.

80| # go to the 80th column

Tar Gzip On-The-Fly

I typically create tarballs by doing "tar cvf file.tar somedir" and then gzipping the resultant tarball. Sometimes there's not enough space on disk to have an intermediate uncompressed file sitting around. As an example, I just made a backup of 48 gigs of MySQL data yesterday. One handy way to avoid storing the uncompressed file on disk is to gzip on the fly.

tar cvf - somedir | gzip -c > somedir.tar.gz

In writing this post, I just noticed that you can also just give tar the -z flag to accomplish the same thing; however, explicitly piping to gzip allows you to specify options to gzip such as compression level.

Change Permissions on All Directories

Sometimes I need to recursively set permissions on all directories in a file hierarchy. Other times, I just want to set file permissions while ignoring directories. Fortunately, the find command makes this easy.

# make all directories read, write, execute for owner. read, execute for group and other
find . -type d -exec chmod 755 {} \;

# make all files read, write for owner. read for group and other
find . -type f -exec chmod 644 {} \;

# make all shell scripts read, write, execute for owner. read, execute for group and other
find . -type f -name "*.sh" 755 {} \;

Vim Can Spell

Thanks to Chris Suter for contributing the following tip:

I always knew this was in there somewhere, but usually my compiler tells me when I'm an idiot, so I've never needed it. Latex, however, does not.

:setlocal spell spelllang=en_us

turns on spell-check and highlighting for incorrectly spelled words.

navigation:
]s - next misspelled word
[s - prev "

interaction:
zg - add misspelled word to system dictionary
zG - add misspelled word to buffer-local set of "good" words
zw - add word to 'bad' list
zW - add word to buffer-local 'bad' list
z= - view a numbered list of suggested spellings from which to choose for auto-replacement

side note: Maybe there's a word list for php fn's, since it seems to assume that when you've mistyped something, you obviously meant 0, "zero", false and NULL simultaneously and chooses arbitrarily between them. This might prevent some headaches/suicides.

SVN: Add All New Files

I've been doing a lot of Ruby on Rails dev work lately, and the framework tends to generate a lot of files. Until recently, I was using a combination of "svn status" piped into grep, awk, and xargs to accomplish adding new files recursively. A friend recently told me to check out the following solution instead.

svn --force add *

The only negative side effect I can see is that it seems to add ignored files back into the tree. This is mostly a non-issue for me, but take note.

S3Virge Woes

I decided to salvage an old junk computer over the weekend. I burned a copy of Xubuntu, and proceeded to install. When the installer tried to initialize X, it couldn't, so I had to do the install in safe graphics mode. The video card, a low-end S3Virge 3d/2x, uses the Xorg s3virge driver. After getting everything working in crappy Vesa mode, I stumbled onto a bug report indicating that there was a bug in s3virge resolution detection that was fixed in Git. I recompiled the appropriate driver from Git, and everything works now. SO, for future reference, here are the steps to re-build the s3virge driver from Git in Xubuntu.

# install packages required for build
apt-get install xorg-dev xserver-xorg-dev libtool autoconf automake

# install git
apt-get install git-core

# check out the repo
git clone git://anongit.freedesktop.org/git/xorg/driver/xf86-video-s3virge

# cd into the repo
cd xf86-video-s3virge

# build and install
./autogen.sh --prefix=/usr && make && sudo make install

Not Vim related I know, but I figure this might help someone.

Window Manipulation

If you have two horizontally split windows and would like to increase the size of the current window, you can do ctrl-w +. A count can be issued beforehand, so to increase the height by 10 lines 10 ctrl-w +. Ctrl-w - will decrease the current window height. To set the current window to an absolute height do N ctrl-w _ where N is the number of lines desired.

In a vertically split scheme, you can use ctrl-w < and ctrl-w > to adjust the width. To set the current window to an absolute width, do N ctrl-w | where N is the number of lines desired.

If you have two windows split horizontally and would like to shift to a vertical scheme, you can do ctrl-w H or ctrl-w L depending on which direction you want the current window to go. Ctrl-w J and ctrl-w K will take a vertically split scheme to a horizontal orientation.