0

there are about 7000000 files in a single directory summing up to a total of 650 GB.

(listing them alone is not even possible)

what options do i have to download these?

i was thinking of using tar to make packages of 1 GB and then downloading these.

or maybe move them in parts to different directories so that its easier to list them.

the OS is ubuntu server.

everytime a file was added, an entry was made into a database and the count there is now >7000000. That also means, i know their filenames.

11
  • How about rsync?
    – Optichip
    Jan 16, 2015 at 12:31
  • 1
    There are around 7000000 different ways you could download these. How about adding some detail about why just downloading them isn't preferable?
    – Paul
    Jan 16, 2015 at 12:40
  • @Paul: if i would use for example ftp, it stucks already when starting to list the files.
    – clamp
    Jan 16, 2015 at 12:42
  • Semi-nitpick: if you can't even list them, how do you know they are there? (If there are files that can't be listed, I for one would assume they aren't there.) Any solution to bundling these files into an archive is going to require listing them.
    – user
    Jan 16, 2015 at 12:51
  • 1
    @MichaelKjörling: cause everytime a file was added, an entry was made into a database and the count there is now >7000000. That also means, i know their filenames, and so i can access them. The files are there.
    – clamp
    Jan 16, 2015 at 12:55

2 Answers 2

1

This is a major PITA:

  • all programs mentioned, everything that uses any variant of ls will crash or time out
  • you must keep track of the progress by other means
  • you must remove stress from the filesystem
  • you must avoid duplicating the situation you are in know. rsync will duplicate the situation. raw tarwill duplicate the situation. you do not want to have 7000000 copies in some other folder.

Ok, given what might work in this situation:

  • pick python or whatever you know best.
  • talk to the database and get filenames in chunks of 1000.
  • copy these 1000 files over to another directory and zip / tar that directory to send it somewhere else. how and if you move these bundled files out of the way depends a bit on the amount of storage you have on that disk. keeping another 7000000 files (no matter how organized) on that same filesystem is stress.
  • if this block of 1000 worked ok and everything is fine, write the highest id for that chunk to a different db-table (you do not want to ALTER TABLE on that 7000000 thing to just add a new colum 'moved', it might take too long and you alter "the truth"; create a 2nd table and track the progress)
  • repeat.

most important thing: copy, verify, then delete the original files in that huge folder.

it's also unwise to 1:1 copy the files somewhere else since you would have repeated the bad situation you are in now.

good luck.

1

If you list the file names in a file called ~/filelist.txt, then go to the directory where all the files are, the following script should move them into manageably-sized sub-directories, one for each day that files were created:-

cat ~/filelist.txt |                                           \
while f=`line`;                                                \
do ( d="`ls -l --time-style=+%y-%m-%d "$f"|awk '{print $6}'`"; \
     [ -d "$d" ] || mkdir "$d";                                \
     mv "$f" "$d"/                                             \
   )                                                           \
done

This will take a long time to run, because you will be running two or three programs for each file, as well as a bash sub-shell.

I suggest that you test it first before you start by using head instead of cat and prefixing echo to the mkdir and mv commands. Of course for this test, mkdir will be listed repeatedly for files with the same date, as the directories are not actually being created. If you repeat with head -n 1000 or more, you should get an idea of how many files will go into each directory.

If there are too many or too few, you can change the time style format to make it weekly or hourly archives. You should aim for around 1000-10000 files in each directory, which will give you 700-7000 directories, both manageable.

Once they are moved into sub-directories, you can zip each directory, which should give you around 7000 files to move, and a small fraction of the 650GB to copy if they're text files.

3
  • ls "$f" is a good candidate for major pain (in this situation). to print the date of a file it's better to just use stat --printf="%z" "$f" | cut -f 1 -d " ". also: the date might be a weak criteria, if the situation was introduced by a bug on one day you will have a lof entries for the same day and thus lots of files end up in the same Y-m-d directory.
    – akira
    Jan 16, 2015 at 15:58
  • 1
    @akira - I'd forgotten about stat, but ls will be OK when passed a single file name. There is a problem only when it has to process the whole directory: if * is passed, the run string will overflow; if called without parameters, sorting 7 million files will take forever and may overflow some buffer. An alternative might be to add a loop counter n and use ((d=n/10000)); printf "%05d\n" $d as the directory name. I didn't suggest this, as there is no guarantee that the files in a directory will have anything in common, though it would be quicker, as no external programs are involved.
    – AFH
    Jan 16, 2015 at 20:30
  • the "counter/10000"-name is the more stable approach, thats als why my answer uses "work in chunks". in the end it depends on the exact situation OP faces.
    – akira
    Jan 16, 2015 at 21:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .