A utility for sorting really big files.

Comments are moderated. It may take a few minutes before your comment appears.
Markdown is supported in your comments.

I also had a problem: insufficient disk space to store even a single uncompressed copy of the data. This makes sorting difficult. Gnu-sort really likes uncompressed data. It wants to read uncompressed input, write uncompressed output and store uncompressed scratch data. Input and output are simple to handle; use pipes and gzip. The scratch space will first use all available ram and spill over into tmpfs if memory use gets too high. This is completely unworkable when your tmpfs is in ram and the uncompressed data is 25 times larger than your ram. There is an option to use disk instead of tmpfs, but that doesn't work either because it will put an entire uncompressed copy onto disk in the process. So it needs to be combined with an obscure option that tells gnu-sort to compress all the scratch data. Overall my best attempt ended up looking like this:

Mail: (not shown)

Please type this: