Linux sort all data in memory

Question

For Linux command sort, how do I force sort to load all input into memory and sort assuming I have enough memory? Or is it best to use a RAMDISK to store the input before feeding it to sort?

Answer 1

Read side

Barring very non-standard filesystems the whole shebang will be read-cached any way (observe this simple in htop).

You can see the amount of buffering as well in vmstat 1 output. Observe how linux will simply take all available memory (even when not addressable to a single client process, e.g. when running a PAE kernel on 32 bit, or 64bit kernel with 32bit userland).

Observe how you can force the cache to be cleared by issueing echo 3 > /proc/sys/vm/drop_caches in another terminal. (clearing page cache, inode and dentry caches)

Write side

On the write side, the tmpfs feature in linux 2.4+ is perfect. It does the analogous of the read caching and you can manually limit it's size. This is my default /tmp mount:

sudo mount -t tmpfs -o nodev,noexec,size=6g none /tmp

I'll usually work on /tmp for longer periods of the day and use version control to push things into a (nonvolatile) repository.

Takeaway

So, shy from /write it yourself/ solutions, you should just use the kernel features that are there.

[1] I also symlink things like ~/.cache ~/.opera/cache etc. into /tmp/ Really lifts the burden of cleaning up, make things fly performance wise and keeps my SSDs in healthy condition

Answer 2

You can specify the temporary directory to be nonexistant and change the main memory size parameter. This will however cause the sort to fail if you don't have enough mem:

$ sort -S 1000 -T /nonexistant/dir /usr/share/dict/words | wc -l 
sort: cannot create temporary file in `/nonexistant/dir': No such file or directory
0
$ sort -S 10000 -T /nonexistant/dir /usr/share/dict/words | wc -l
98569

Unit for the -S option is kB (see the comment below).

Answer 3

If whichever temporary directory "sort" uses supports "delayed allocation" - for example, ext4 or xfs, then it will do this anyway!

If the kernel decides there is enough memory, it won't need to write the data to disc; if the sort happens sufficiently quickly that the kernel doesn't write the data to disc, then the files are deleted, they will be thrown away and forgotten about without any writes at all.