Thursday, October 31, 2013

Cheap Process Monitoring: Echo, Watch, and Netcat

I frequently work with memcached, gearman, and a variety of other services that allow you to retrieve useful stats by connecting via telnet and issuing a status command. In a lot of cases, retrieving these stats in realtime is beneficial for debugging and diagnosing data pipeline issues.

Fortunately, a few basic tools found on almost any Linux system can make this easy.

The watch utility runs whatever command you provide it at the interval you specify. Here's the world's cheapest realtime clock.
$ watch -n 1 'date'
Netcat is too versatile to cover in depth in this post, but one of its most basic capabilities is accepting data via standard input, sending that data to a host / port combination of your choosing, and then dumping the result to standard output.
$ printf "HEAD / HTTP/1.0\r\n\r\n" | nc 80 | grep Server
Server: gws
The above command takes the HTTP HEAD request from printf, sends it to netcat, which then passes it on to on port 80. The results are printed to standard output, and the grep command limits the result output to the server returned (in this case gws).

Combining these two commands provides a ton of utility. Here are two useful examples that I use regularly.

1) Monitoring Gearman queues in realtime
watch -n 1 '(echo status; sleep 0.1) | nc localhost 4730'

publishMessage  4       0       1
consumeMessage  7       0       8
archiveMessage  28      0       12
2) Monitoring memcached get/set commands in realtime
watch -n 1 '(echo stats; sleep 0.1) | nc localhost 11211 | grep cmd'

STAT cmd_get 506
STAT cmd_set 8
STAT cmd_flush 0
STAT cmd_touch 0
STAT auth_cmds 0

No comments: