Wednesday, April 8, 2009

bashreduce

I stumbled across this project recently. It's a program that allows you to apply standard shell tools in a map reduce style fashion. Large memory hungry tasks like sorting can be distributed across machines to maximize throughput. As a bonus, it's written in bash, so it's highly portable and works with a combination of vanilla shell utilities and ssh shared-key authentication.

The bashreduce project.

No comments: