I stumbled across this project recently. It's a program that allows you to apply standard shell tools in a map reduce style fashion. Large memory hungry tasks like sorting can be distributed across machines to maximize throughput. As a bonus, it's written in bash, so it's highly portable and works with a combination of vanilla shell utilities and ssh shared-key authentication.
The bashreduce project.
skip to main |
skip to sidebar
A blog dedicated to text editing and general exploration of computing knowledge
Posts
No comments:
Post a Comment