GZ-Sort

A utility for sorting really big files.

Comments are moderated. It may take a few minutes before your comment appears.
Markdown is supported in your comments.

You mean to say that after all that, gnu-sort is still faster? Sadly yes. Gz-sort did use slightly less scratch space (410MB vs 500MB). But I feel that I am missing something. At that level of disk usage, gnu-sort should have been able to process the Freebase data. Gnu-sort appears to have yet another level of "gotcha" built in, beyond the default tmpfs and uncompressed scratch gotchas, that crops up at these scales. Gz-sort wins in the "does not do anything surprising" category, scales perfectly over truly monstrous files, and actually manages to finish the big job.

Name:
Mail: (not shown)

Please type this: