1

I'm using Linux merge sort (sort -m) to sort a large group of presorted files. I was assuming that merge sort would be considerably faster than a full sort on the unsorted data but that's not what I'm seeing. For example, if I do a full sort on 1000 files containing 10000 lines of unsorted data:

/bin/time -o ./full_sort.time -v sort --parallel 8 -k1,1g ./data/unsorted/f* > /dev/null

I see the following times:

User time (seconds): 104.70
System time (seconds): 0.71
Percent of CPU this job got: 440%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:23.90

Now, if I presort the data and run the exact same sort while invoking the -m option:

/bin/time -o ./merge_sort.time -v sort --parallel 8 -k1,1g -m ./data/presorted/f* > /dev/null

I see these times:

User time (seconds): 59.34
System time (seconds): 2.15
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:01.60

The merge sort wall time is actually slower than the full sort! Again, I was expecting the merge sort to be considerably faster since each input file in the data set is presorted and only the merge step remains to be done. There must be a something that I'm missing. Can someone please clarify what I'm doing wrong?

Thanks, JCS

New contributor
JCS_ATX is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
  • Does this fix the performance of merge? /bin/time -v sort -k1,1g --batch-size=1000 -m ./data/presorted/f* > /dev/null ? – filbranden 2 days ago

Your Answer

JCS_ATX is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.