Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

For Linux command sort, how do I force sort to load all input into memory and sort assuming I have enough memory? Or is it best to use a RAMDISK to store the input before feeding it to sort?

share|improve this question

migrated from stackoverflow.com Jun 11 '11 at 19:49

This question came from our site for professional and enthusiast programmers.

    
"sort all data in memory": something like od /dev/mem -An | sort came to mind immediately –  sehe Jun 11 '11 at 19:00
    
What are you trying to achieve? If you want sort to be as fast as possible, let it do what it wants to do. If you don't want it to access the filesystem, don't give it a filesystem to access, as shown by viraptor. I'm having trouble coming up with a use case though. –  Gilles Jun 11 '11 at 19:08
add comment

3 Answers

You can specify the temporary directory to be nonexistant and change the main memory size parameter. This will however cause the sort to fail if you don't have enough mem:

$ sort -S 1000 -T /nonexistant/dir /usr/share/dict/words | wc -l 
sort: cannot create temporary file in `/nonexistant/dir': No such file or directory
0
$ sort -S 10000 -T /nonexistant/dir /usr/share/dict/words | wc -l
98569

Unit for the -S option is kB (see the comment below).

share|improve this answer
2  
The unit for -S is kB unless you add a suffix, e.g. -S 1000 = 1024000 bytes, -S 1000b = 1000 bytes, -S 1% = 1% of physical memory. See the description of SIZE just below the option list, or the hypertext manual. –  Gilles Jun 11 '11 at 19:06
    
Note that is is the amount of memory used not the amount of input sorted at once. If you are sorting a 2GiB file you will need more than -S2G. –  Kevin Cox Dec 14 '13 at 0:35
add comment

Read side

Barring very non-standard filesystems the whole shebang will be read-cached any way (observe this simple in htop).

You can see the amount of buffering as well in vmstat 1 output. Observe how linux will simply take all available memory (even when not addressable to a single client process, e.g. when running a PAE kernel on 32 bit, or 64bit kernel with 32bit userland).

Observe how you can force the cache to be cleared by issueing echo 3 > /proc/sys/vm/drop_caches in another terminal. (clearing page cache, inode and dentry caches)

Write side

On the write side, the tmpfs feature in linux 2.4+ is perfect. It does the analogous of the read caching and you can manually limit it's size. This is my default /tmp mount:

sudo mount -t tmpfs -o nodev,noexec,size=6g none /tmp

I'll usually work on /tmp for longer periods of the day and use version control to push things into a (nonvolatile) repository.

Takeaway

So, shy from /write it yourself/ solutions, you should just use the kernel features that are there.

[1] I also symlink things like ~/.cache ~/.opera/cache etc. into /tmp/ Really lifts the burden of cleaning up, make things fly performance wise and keeps my SSDs in healthy condition

share|improve this answer
    
All Linux distros already have a writable tmpfs mounted on /dev/shm, which can be abused for sorting. Some now have /run, too. –  grawity Jun 12 '11 at 8:55
    
@grawity, But tmpfs has a set size, so you can run out of "disk space" on your ram disk. For example I have 4GiB of ram but sorting a 3GiB file fails because I only have a 2GiB ram disk. It is also less efficient because it has to worry about "writing it out" and "reading it back". –  Kevin Cox Dec 6 '13 at 3:13
    
@KevinCox It has a maximum size, yes. But not all systems use a tmpfs for /tmp; when 'sort' was written, tmpfs did not even exist. And there is another directory, /var/tmp, which is explicitly required by FHS not to be a tmpfs and programs can use it for storing huge files. // As for writing data out, don't forget mmap –  grawity Dec 6 '13 at 11:43
1  
@KevinCox You can just on-the-fly resize. On my systems I have sometimes done mount -o remount,size=24g /tmp - and voila :) You can also size back down (assuming you free the space first) –  sehe Dec 6 '13 at 12:04
add comment

If whichever temporary directory "sort" uses supports "delayed allocation" - for example, ext4 or xfs, then it will do this anyway!

If the kernel decides there is enough memory, it won't need to write the data to disc; if the sort happens sufficiently quickly that the kernel doesn't write the data to disc, then the files are deleted, they will be thrown away and forgotten about without any writes at all.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.