0

I'm sorting a huge file, around 400 gigabytes. I'm running out of the disk space and I must do something quickly.

Let's assume the original file is called original_file. So I execute (simplified) it as "sort original_file | gzip -c > output_file"

I use /home/tmp as a temporary dir. From what I see, there are a lot of intermediate files, like so:

tmpA465

tmpB154

...

and so on.

The smallest ones have size 12 megabytes.

The largest have ~182 megabytes.

So, it seems that the "sort" command have already split the original file into small pieces, and have sorted them, and now it is merging them into bigger parts (which will be, eventually, sorted as well). Please correct me if I'm wrong.

Can I remove the original file right now without terminating the sort process? I've been waiting for a few days for that and it's important that the "sort" command will not fail and I will get the result file, finally.

The OS is Ubuntu server 13.04, x64.

Thanks!

1
  • Did you try to use parallel option? Does it function for you? Let me know
    – Hastur
    Jun 4, 2014 at 10:32

1 Answer 1

0

If you have a recent version of sort (8.11+) you can speed up the process with sort --parallel=N option: you will share the work on N cores. This can really boost your job.

sort --parallel=N 

For what if concerns sort algorithm you should see e.g. Algorithmic details of UNIX Sort command.
It uses the a variation some Merge sorting: it means that it sorts separating in block the whole work, and after merge the sorted blocks. You can find pieces on /tmp.

It reads pieces that fit in memory. If you try to sum the size of all the tmp pieces that you find, you will obtain a value not necessary related to the percentage of work done. But if it is a lot less you can deduce it still need you file.

I'm afraid that you cannot know the size of output processed till the last passage. The key point should be in the last sort passage when it will merge the 2 half just sorted. It can proceed creating the first half and after the second one. Or it can proceed processing in parallel the two halves. But in the 1st case it will use less space on the hard disk. So I suppose you cannot delete the file.

To have a final answer to your doubt you should see the code downloading your version of sort from gnu or looking from git.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .