9

I'm trying to make unique a set of lines pulled from a file with egrep with sort -u, then count them. About 10% of the lines(all 100 characters long from the alphabet [ATCG]) are duplicated. There are two files, about 3 gigs each, 50% aren't relevant, so perhaps 300 million lines.

LC_ALL=C  grep -E  <files> |  sort --parallel=24  -u | wc -m

Between LC_ALL=C and using -x to accelerate grep, the slowest part by far is the sort. Reading the man pages led me to --parallel=n, but experimentation showed absolutely no improvement. A little digging with top showed that even with --parallel=24, the sort process only ever runs on one processor at a time.

I have 4 chips with 6 cores and 2 threads/core, giving a total of 48 logical processors. See lscpu because /proc/cpuinfo would be too long.

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                48
On-line CPU(s) list:   0-47
Thread(s) per core:    2
Core(s) per socket:    6
Socket(s):             4
NUMA node(s):          8
Vendor ID:             AuthenticAMD
CPU family:            21
Model:                 1
Stepping:              2
CPU MHz:               1400.000
BogoMIPS:              5199.96

What am I missing? Even if the process is IO-bound, shouldn't I see parallel processing anyway? The sort process uses 99% of the processor it is actually on at any given time, so I should be able to see parallelization if it's happening. Memory isn't a concern, I have 256 Gb to play with and none of it is used by anything else.

Something I discovered piping grep to a file then reading the file with sort:

 LC_ALL=C  grep -E  <files>  > reads.txt ; sort reads.txt  -u | wc -m

default, file 1m 50s
--parallel=24, file 1m15s
--parallel=48, file 1m6s
--parallel=1, no file 10m53s
--parallel=2, no file 10m42s
--parallel=4 no file 10m56s

others still running

In doing these benchmarks it's pretty clear that when piped input sort isn't parallelizing at all. When allowed to read a file sort splits the load as instructed.

  • What sort is that on which distribution? The standard sort doesn't know that option. – ott-- Jul 9 '15 at 20:17
  • uname -a gives "3.13.0-46-generic #79-Ubuntu SMP" and lsb_release -a claims 14.04.2 codename trusty , and the version of sort that's part of the gnu coreutils, according to man sort. – Jeremy Kemball Jul 9 '15 at 20:22
  • It seems to me there are portions here that needs to be re-read: gnu.org/software/coreutils/manual/html_node/… – Hannu Jul 9 '15 at 20:29
  • I'm not sure I understand what you're getting at @Hannu, could you be more specific? sort --parallel=2 doesn't parallelize either. Neither does 4 or 8. nproc gives back 48 like it should. – Jeremy Kemball Jul 9 '15 at 20:36
  • 1
    I'd say... don't use coreutils for this. Amusingly we had a very similar question and well.... every other method works better superuser.com/a/485987/10165 – Journeyman Geek Jul 10 '15 at 2:23
21

sort doesn't create a thread unless it needs to, and for small files it's just too much overhead. Now unfortunately sort treats a pipe like a small file. If you want to feed enough data to 24 threads then you'll need to specify to sort to use a large internal buffer (sort does that automatically when presented with large files). This is something we should improve on upstream (at least in documentation). So you'll want something like:

(export LC_ALL=C; grep -E  <files> | sort -S1G --parallel=24 -u | wc -m)

Note I've set LC_ALL=C for all processes, since they'll all benefit with this data).

BTW you can monitor the sort threads with something like:

watch -n.1 ps -C sort -L -o pcpu

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.