Copying large number of files from one directory to another in Linux

Question

I've a directory containing around 2.8 lacs of files. I want to move them to another directory.

If I use cp or mv then I get an error 'argument list too long'.

If I write a script like

for file in ls *; do
   cp {source} to {destination} 
done

then because of ls command , its performance degrades.

How can I do this?

Answer 1

Use rsync:

$ rsync -a {source}/ {destination}/

e.g.

$ rsync -a /some/path/to/src/ /other/path/to/dest/

(note the trailing /s)

Answer 2

How about when moving (instead of copying):

$ find {origin}/ -maxdepth 1 -name "*" -o -name ".*" -exec mv '{}'  {destination}/ ';'

I think that will move keeping the structure (subdirs) and hidden files or dirs, plus no extra space consumed as with rsync + rm. And if {origin} and {destination} are in the same partition it will be faster.

Answer 3

You don't need the ls, you can simply use

for file in *; do
    cp $file /your/dest
done

or you can do something like:

echo * | xargs -i cp {} /your/dest

Answer 4

#!/bin/bash
d=$(date +%Y%m%d%H%m%s)
cd /path
tar zcvf "/destination/bakup_${d}.tar.gz" mydirectory_for_transer

Answer 5

The problem is the * expansion which fails because there are too many entries. a safe workaround which always works is instead of using ls * is simply using ls (thus not using the * expansion (which is done by your shell, not by the ls command)

for i in ls; do
   cp $i $dest
done 

Or offcourse the ls | xargs or the find solution presented elsewhere.

Answer 6

Assuming you want to move the files within the same filesystem, you could just rename the directory containing your lacs and be done with it.

Answer 7

I like rsync for this, or:

find dir1 -type f -exec cp {} dir2 \;