Brace Expansion

A little known feature of bash is brace expansion. Brace expansion expands the items within the braces to the arguments adjacent to the brace.

Quick example:

echo foo{bar,car}
# yields
foobar foocar

What's the point?

You can use brace expansion in your shell scripts or for quick one-liners. An example:

cp foo.c{,.blah} # backup foo.c to foo.c.blah

Sort + De-dupe? Easy.

Sometimes when dumping data, it makes more sense from a performance perspective to not worry about removing duplicate data when constructing a SQL query and do it after the fact instead. This is accomplished really easy on the shell as follows:

cat filename.csv | sort --buffer-size=32M | uniq > filename_uniq.csv

You can omit the buffer-size argument to sort in favor of the default size or set it to whatever you want.

Screen Tips

I use screen quite a bit for a number of reasons. I plan to periodically edit this blog entry as a notepad for GNU screen features I use frequently. If you're not familiar with screen, at the most basic functionality, it lets you do things like run multiple processes inside of one terminal. These processes are typically shells, but I frequently run a split session between Vim and a shell (see screenshot below). It also allows you to detach from your current session and later return. This is great for things like long-running compiles, keeping tabs on a process, or logging out of an ssh session without killing the foregrounded process. Read below for a summary of all these features (and more).

Basics:

At the basic level, you want to fire up screen and give it a session name. This is useful for coming back and re-attaching to the session later. Start screen like this:

screen -S dailyvim

You'll be presented with an identical looking shell that you were in prior to the command, but don't be fooled. Type in a command like "ls -l" and then hit ctrl-a c (create screen). You'll be presented with a new shell with an empty command line. What's happened is you're actually running two shells now within a single terminal. Type in a new command like "date" then hit ctrl-a n (next screen). You should be taken back to the directory listing you saw before. Now hit ctrl-a d (detach). The entire session is pushed to the background, and you can resume it whenever you would like. If you want, you can close the current term, open a new one, or even do a remote login and still resume the session. Re-attach to your screen session by issuing the following command, screen -R dailyvim.

Scrollback, Copy & Paste:

Another nice feature of screen is that it allows you to scrollback through your shell session as needed to look at information that may have scrolled by too quickly to read. The length of the scrollback buffer is managed through your .screenrc file, but for most purposes the default is probably ok. Anytime you're inside of screen, scrollback can be invoked via Ctrl-a ESC. Once in scrollback mode, Vi navigation keys conveniently allow you to scroll around; although, you can also use the arrow keys if desired. From inside scrollback, you can yank text into a copy buffer if you so desire. Press the space bar once to initiate a copy and press it again to complete the copy. To paste the copied text press Ctrl-a ]. The following Vi commands work in this mode (from TFM):

h, j, k, l move the cursor line by line or column by column.
0,^ and $ move to the leftmost column, to the first or last non-whitespace character on the line.
H, M and L move the cursor to the leftmost column of the top, center or bottom line of the window.
+ and - positions one line up and down.
G moves to the specified absolute line (default: end of buffer).
| moves to the specified absolute column.
w, b, e move the cursor word by word.
B, E move the cursor WORD by WORD (as in vi).
C-u and C-d scroll the display up/down by the specified amount of lines while preserving the cursor position. C-b and C-f scroll the display up/down a full screen.
g moves to the beginning of the buffer.
% jumps to the specified percentage of the buffer.

Split Window:

Sometimes I like to run a shell underneath my Vim session. Screen makes this possible by allowing me to do a horizontal split on the current screen session. From inside screen, hit Ctrl-a S (capital). You should now have two windows. Hop between them by pressing Ctrl-a tab. The second window is probably blank. Give it focus and hit Ctrl-a c (create) to create a session inside of it. You can do Ctrl-a n (next) as usual to cycle through sessions.

Command Mode:

Screen provides a slew of commands that can manipulate behavior of your current session. To give a command, you need to initially enter command mode. To do this, type Ctrl-a :. From there you can type in any command screen recognizes. I use command mode to resize a split window to my desired height as follows (Ctrl-a :resize N) where N is the number of rows high you want to current window. It also supports plus and minus arguments on N for relative adjustment.

Other tricks:

hardcopy command: From inside screen, type Ctrl-a h (hardcopy), and screen will dump whatever is displayed in the current buffer to a file "hardcopy.n". This is handy if you don't want to (or can't) use the mouse to copy and paste.

digraph mode: Ctrl-a Ctrl-v allows you to enter digraphs on the command line. Try :digraph in Vim for a listing of available digraphs.

list sessions: screen -ls lists all sessions currently associated with your UID.

re-attach: screen -d will attach to a given session (even if currently attached). This is useful if you forgot to detach before you left the office, etc...

list windows: Ctrl-a " will give you a selectable list of windows in the current session.

name window: Ctrl-a A will let you name the current window for the window list.

currently attached: Ctrl-a * will show currently attached displays.

Piping Buffers

Say you have a buffer in Vim, and you want to run it through an external command. For the sake of example, let's say that command is sort. All you have to do is the following:

:% ! sort

or more generally

:% ! command

If you want to bring the output of a command into buffer you can do:

:r ! ls -l

or even:

:r ! cat filename | sort

Perl Mass Substitution

Here's a nice trick I've used many many times over the years. Say that you have a group of files, and you want to replace some text in every file quickly and easily. Perl makes this really easy, and it's ubiquity pretty much guarantees it's around if you're on a *nix platform. For the sake of example, we'll pretend we have a group of html files, and we want to change every occurrence of index.html to index.php. The syntax is as follows:

[edited - thanks Chris!]

perl -pi.bak -e 's/index\.html/index.php/gi' *.html

The command line options provided break down as follows:

-p assumes a while loop over each line of every file and implicitly prints
-i specifies that you want in-place substitution on the file (no redirection of STDOUT required)
-i also takes an option argument of a backup file extension (.bak in this case)
-e tells perl to run the following code on the command line

From there, you're using standard Perl regular expressions. Don't worry if you're not a regexp guru. In the simplest form, you can simply replace one string with another.

The Perl code breaks down as follows:

s(means substitute)/string you want to match/string to replace/
(g means replace multiple instances per line)
(i means case-insensitive matching... aka ignore uppercase and lowercase differences)

Of course if you know regular expressions, you can do all kinds of fancy stuff.

Here's a nice reference.

Drag Windows From Anywhere

Here's a basic X windows tip. From any window manager, if you hold down the Alt key, you can drag a window from any position (the middle, the top, the bottom, etc...). This is handy when you don't want to move the mouse all the way up to the top toolbar to drag.

PHP Test Trick

This is a trick that works in a lot of languages. Say you have a file containing a library that also has some utility in it's own right. As an example, I wrote a metadata parser at Grooveshark called MetaShark. It's basically a file containing a bunch of "utility functions" that are invoked via a call to main() inside of the file. It came time to write a unit test for the parser, but I didn't want to call main() from inside of the test suite because that would have the unwanted side effect of processing the current batch of jobs. Instead I want to test the methods individually and make MetaShark just work as a library. All you have to do to make this work is something simple like this:

if ($argv[0] == basename(__FILE__)) {
main();
}

or in Ruby:

if __FILE__ == $0
main()
end

Extrapolate to your language of choice.

Remote Port Forwarding over SSH

Consider the following scenario. You have a remote machine that you have access to, and you have enough control to bind a process to an arbitrary port. You also have a workstation that isn't exposed to the outside world, but you would like to access this workstation from outside of your LAN/firewall. Sure you can setup a VPN, configure port forwarding on your router, etc... As usual, there are lots of solutions, but not many are as cool as this quick little ssh hack.

On your workstation:

ssh -Nf -R 9000:localhost:22 username@publicserver.com

This assumes you can bind to port 9000 via the loopback on the remote server and that you're running sshd on port 22 on your local workstation. Now you drive to the local coffeehouse, and you need to hop on your Linux box at home...

Log into publicserver.com:

ssh -p 9000 username@localhost

In this case, username is your account on your workstation behind the firewall.

Thanks to Nate G. for this tip.

Tar + SSH

If you want to transfer a directory structure from your local machine to a remote host, there are obviously a lot of ways to do this. You could use a recursive scp, an rsync, ftp (god forbid), or a variety of other techniques. Another way that's assuredly less popular but has the advantage of letting you specify the compression method and providing options to preserve attributes verbatim, is to use tar and ssh with the following syntax:

tar cvjf - * | ssh whoever@machine.com "(cd /path; tar xjf -)"

The previous example uses bzip2 compression, which may save time for large transfers. Thanks to Nate G. for contributing this tip.

Reading STDIN

This is pretty basic, but it's also really useful. Vim can read standard input via a pipe at the end of any common command chain. I use Vim in conjunction with diff all the time to analyze the difference between two or more source files.

diff -u Scheduler.old.java Scheduler.java | vim -

Compare the difference of two source trees before submitting your patch:

diff -Naur oldtree newtree | vim -

Vi Compatible Shell

I like to use zsh as my standard shell for a number of reasons, but I realize the majority of Linux users use bash and Unix users usually use csh or tcsh. Either way, you can allow Vi compatible keybindings in any of these very simply:

in bash:
set -o vi

in zsh, csh, tcsh:
bindkey -v

From there, you can navigate the command-line using Vi key bindings. You can also use most common motions and operators to edit your commands.

Using xargs

I'm a big fan of the xargs command. It's widely used in shell scripts but not so much on the command line. It's a really handy command when you want to take an incoming argument and apply a command to it. Here's an example that would kill every process matching a given pattern of "firefox" (similar to killall).

ps aux | grep firefox | awk '{print $2}' | xargs kill -9

remove files matching *lock*:

ls *lock* | xargs rm

for more complex commands, you can set the incoming data to be stored in a token:

ls *lock* | xargs -I $ ls -l $

or

ls *lock | xargs -I {} ls -l {}

Obviously the possibilities go far beyond what's shown here. Play around with it.

Expanding Scope

Just a quick note: I'm going to be expanding the scope of this blog a bit to not only include tips strictly related to Vim but also shell magic, Linux tips, etc... The American Heritage dictionary has Vim defined as:

vim (vĭm)
n.

Ebullient vitality and energy. See synonyms at vigor.

So, I'll be applying ebullient vitality and energy to trying to make this blog interesting. Also, FIY, I will still be posting Vim tips with the same frequency so no worries there.

SCP'ing

You can use Vim to edit a file on a remote machine running ssh/scp very easily.

vim scp://username@hostname.com:port//some/file

You have to authenticate as usual, but it can be handy for a quick hack. Also, other protocols besides scp are supported. Try :help scp for more info.

Tree Explorer

A lot of times I like to run Vim in a single window with many buffers. A nice way to stay focused is to go full-screen and then run the vtreexplorer plugin on the root of my source tree. A simple entry in my vimrc, and I can toggle it on as needed:

map <F12> :VSTreeExplore <CR>

VSTreeExplorer is available at: http://vim.sourceforge.net/scripts/script.php?script_id=184

Here's the obligatory screenshot.

Delete Lines Matching Keyword

Neat little trick using the global ex command to remove all lines matching a given pattern.

:g/.*foo.*/d

Of course you can use any ex command at the end and you can put in whatever pattern you'd like.