Cheap Process Monitoring: Echo, Watch, and Netcat

I frequently work with memcached, gearman, and a variety of other services that allow you to retrieve useful stats by connecting via telnet and issuing a status command. In a lot of cases, retrieving these stats in realtime is beneficial for debugging and diagnosing data pipeline issues.

Fortunately, a few basic tools found on almost any Linux system can make this easy.

The watch utility runs whatever command you provide it at the interval you specify. Here's the world's cheapest realtime clock.

$ watch -n 1 'date'

Netcat is too versatile to cover in depth in this post, but one of its most basic capabilities is accepting data via standard input, sending that data to a host / port combination of your choosing, and then dumping the result to standard output.

$ printf "HEAD / HTTP/1.0\r\n\r\n" | nc google.com 80 | grep Server
Server: gws

The above command takes the HTTP HEAD request from printf, sends it to netcat, which then passes it on to google.com on port 80. The results are printed to standard output, and the grep command limits the result output to the server returned (in this case gws).

Combining these two commands provides a ton of utility. Here are two useful examples that I use regularly.

1) Monitoring Gearman queues in realtime

watch -n 1 '(echo status; sleep 0.1) | nc localhost 4730'

publishMessage  4       0       1
consumeMessage  7       0       8
archiveMessage  28      0       12
.

2) Monitoring memcached get/set commands in realtime

watch -n 1 '(echo stats; sleep 0.1) | nc localhost 11211 | grep cmd'

STAT cmd_get 506
STAT cmd_set 8
STAT cmd_flush 0
STAT cmd_touch 0
STAT auth_cmds 0

Sort: Human Numeric

The sort command can accept input data formatted in human readable form. The du -h command is a longtime favorite for finding file sizes in a format that's easy to read with the naked eye. Coupled with sort, you can easily find and display the largest files in a directory tree.

$ du -h | sort -r --human-numeric | head

Update: I should have mentioned that --human-numeric is a recent addition to GNU coreutils, so this won't work on OS X or older distros. Here's a more generic solution (borrowed from here).

for i in G M K; do du -ah | grep [0-9]$i | sort -nr -k 1; done | head -n 11

Handy Shell Pattern

When piping commands, you can often benefit from breaking your data into multiple lines. A combination of sed and xargs can make this very easy to do. Take the following commands for example:

cd /tmp && mkdir blah && cd blah;
touch file-1 file-2 file-3;
ls | sed 'p; s/-//' | xargs -n2

Breaking this down, we create a directory and touch three empty files. From there, we use sed to print two lines. The first line is the original file name, the second line is the filename without the dash. From here, we can use the -n argument in conjunction with xargs to merge the two lines back into a single line. The resulting output would be:

file-1 file1
file-2 file2
file-3 file3

Once you have the lines joined, you could do something like:

... | while read a b; do mv $a $b; done

Bonus tip: the "read" bash builtin takes a line and associates each word with a given variable. Type "help read" from the shell for the complete documentation.

In summary the full command-chain for this tip would be:

cd /tmp && mkdir blah && cd blah;
touch file-1 file-2 file-3;
ls | sed 'p; s/-//' | xargs -n2 | while read a b; do mv $a $b; done

Thanks to Chris Sutter for contributing this handy shell pattern, and thanks to Andeers for the coffee!

Quickly Find Subdir

I used to use a long-winded invocation of find with the prune option to non-recursively find all the directories in the current dir:

find . -name . -o -type d -print -prune

From there, I moved to using ls and grep:

ls -l | grep '^d'

I recently learned that the following shell pattern works just as well:

ls -ld */.


Google Shell

I haven't played around with this very extensively, but I'm a fan of just about any sort of shell, so I thought this was sort of fun.

A non-google sponsored Google Shell.

Brace Expansion

A little known feature of bash is brace expansion. Brace expansion expands the items within the braces to the arguments adjacent to the brace.

Quick example:

echo foo{bar,car}
# yields
foobar foocar

What's the point?

You can use brace expansion in your shell scripts or for quick one-liners. An example:

cp foo.c{,.blah} # backup foo.c to foo.c.blah

Sort + De-dupe? Easy.

Sometimes when dumping data, it makes more sense from a performance perspective to not worry about removing duplicate data when constructing a SQL query and do it after the fact instead. This is accomplished really easy on the shell as follows:

cat filename.csv | sort --buffer-size=32M | uniq > filename_uniq.csv

You can omit the buffer-size argument to sort in favor of the default size or set it to whatever you want.

Vi Compatible Shell

I like to use zsh as my standard shell for a number of reasons, but I realize the majority of Linux users use bash and Unix users usually use csh or tcsh. Either way, you can allow Vi compatible keybindings in any of these very simply:

in bash:
set -o vi

in zsh, csh, tcsh:
bindkey -v

From there, you can navigate the command-line using Vi key bindings. You can also use most common motions and operators to edit your commands.

Using xargs

I'm a big fan of the xargs command. It's widely used in shell scripts but not so much on the command line. It's a really handy command when you want to take an incoming argument and apply a command to it. Here's an example that would kill every process matching a given pattern of "firefox" (similar to killall).

ps aux | grep firefox | awk '{print $2}' | xargs kill -9

remove files matching *lock*:

ls *lock* | xargs rm

for more complex commands, you can set the incoming data to be stored in a token:

ls *lock* | xargs -I $ ls -l $

or

ls *lock | xargs -I {} ls -l {}

Obviously the possibilities go far beyond what's shown here. Play around with it.