How can I use GNU's coreutils split command to split a large file into pieces of L number of lines each, but putting each of the different subfiles into different directories at the same time they are created?

Right now I have a perl script that creates a hierarchy of directories starting from ./00/00/00 to ./99/99/99 and doing perl prints of each line into subfiles in the script, but it's much slower than GNU's split command would do, creating all the subfiles in one directory. I would like to have them in the hierarchy of directories since it makes things more manageable for me later on.

Any ideas?

  • Will you satisfy if something just sorts files into subfolders after split will split your file? – rush Jul 2 '12 at 12:58
  • It would be great if it could be done while split is working. – 719016 Jul 2 '12 at 13:50

The program does not support this (according to the info and man pages). You best bet is to move the files after they are created. Your next best bet is to add this feature and mail a patch to the current maintainer.

The first option is really not that hard to do and should not be very slow unless you are on a network mounted share. Consider using a fake reiserfs partition for this:

dd if=/dev/zero of=folders.reserfs bs=1M seek=5000 count=0
losetup  /dev/loop7  folders.reserfs
cfdisk /dev/loop7 # (make 1 primary partition)
mkreiserfs /dev/loop7
mkdir folders
sudo mount /dev/loop7  folders/  -o loop
cd folders

$ time echo  {00..99}/{00..99}/{00..99} |tr ' ' '\n' | xargs -L 10000 mkdir -p
time: Real 1m28.2s  User 0m9.9s  System 1m12.0s

move the files with something like (probably wiser to move as many files as possible at once and not 1 by 1 - use wildcards and xargs -L):

for i in x*; do echo mv $i ${i:1:1}/${i:2:2} ;done
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.