Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a directory containing around 280,000 files. I want to move them to another directory.

If I use cp or mv then I get an error 'argument list too long'.

If I write a script like

for file in ls *; do
   cp {source} to {destination} 
done

then, because of the ls command, its performance degrades.

How can I do this?

share|improve this question

migrated from stackoverflow.com Feb 11 '10 at 8:59

This question came from our site for professional and enthusiast programmers.

    
What is the total size of all files? Maybe first tar these files? – skwllsp Feb 10 '10 at 14:16
    
See this question. – Nick Presta Feb 10 '10 at 14:20

Use rsync:

$ rsync -a {source}/ {destination}/

e.g.

$ rsync -a /some/path/to/src/ /other/path/to/dest/

(note the trailing /s)

share|improve this answer
    
bash: /usr/bin/rsync: Argument list too long Sorry Paul!!!!! – Ritesh Sharma Feb 10 '10 at 15:13
    
@Ritesh -- I'm guessing you specified some files or * as part of {source} - it should just be a directory, e.g. rsync -a /some/path/src/ /other/path/to/ -- note the trailing /s. – Paul Feb 10 '10 at 15:38
    
Yes Paul. I gave the path of the directory. but it didn't work! – Ritesh Sharma Feb 11 '10 at 4:08
    
@Ritesh - that doesn't seem possible - can you copy and paste the actual rsync command and resulting error message(s) from your terminal ? – Paul R Feb 12 '10 at 8:41

I am missing two twings in the answers here, so I am adding yet another one.

Though this reminds me of adding yet another standard answer...

enter image description here

There are two problems here:

I have a directory containing around 280,000 files.

Most tools do not scale all that well with this number of files. Not just most Linux tools or windows tools, but quite a lot of programs. And that might include your filesystem. The long term solution would be 'well, do not do that then'. If you have different files, but them in different directories. If not expect to keep running into problems in the future.

Having said that, lets move to your actual problem:

If I use cp or mv then I get an error 'argument list too long'

This is caused by expansion of * by the shell. The shell has limited space for the result and it runs out. This means any command with an * expanded by the shell will run into the same problem. You will either need to expand fewer options at the same time, or use a different command.

One alternate command used often when you run into this problem is find. There are already several answers showing how to use it, so I am not going to repeat all that. I am however going to point out the difference between \; and +, since this can make a huge performance difference and hook nicely into the previous expansion explanation.

find /path/to/search --name "*.txt" -exec command {} \;

Will find all files under path/to/search/ and exec a command with it, but notice the quotes around the *. That feeds the * to the command. If we did not encapsulate it or escape it then the shell would try to expand it and we would get the same error.

Lastly, I want to mention something about {}. These brackets get replaced by the content found by find. If you end the command with a semicolom ; (one which you need to escape from the shell, hence the \;'s in the examples) then the results are passed one by one. This means that you will execute 280000 mv command. One for each file. This might be slow.

Alternatively you can end with +. This will pass as many arguments as possible at the same time. If bash can handle 2000 arguments, then find /path -name "*filetype" -exec some_move {}+ will call the some_move command about 140 times, each time with 2000 arguments. That is more efficient (read: faster).

share|improve this answer

You don't need the ls, you can simply use

for file in *; do
    cp $file /your/dest
done

or you can do something like:

echo * | xargs -i cp {} /your/dest
share|improve this answer
    
The first solution won't work coz of performance issue but I should try the second one. I'll but after some time. Thanks. – Ritesh Sharma Feb 10 '10 at 14:24
    
The first solution didn't work for me either. This one is the only one that worked. – Whitecat Jun 19 '15 at 20:45
    
The first solution lacks proper quoting, but other than that, it should work, and be better than the second. Proper quoting means double quotes around "$file" inside the loop. – tripleee Apr 25 at 14:10
#!/bin/bash
d=$(date +%Y%m%d%H%m%s)
cd /path
tar zcvf "/destination/bakup_${d}.tar.gz" mydirectory_for_transer
share|improve this answer
    
I think I should go for this. But one question still ticks in my mind and that is performance? – Ritesh Sharma Feb 10 '10 at 14:29
1  
i do not have a million files to test, so i can't answer for you about performance. you have to test out yourself on a development server. – user31894 Feb 11 '10 at 3:21

Assuming you want to move the files within the same filesystem, you could just rename the directory containing your lacs and be done with it.

share|improve this answer

I like rsync for this, or:

find dir1 -type f -exec cp {} dir2 \;
share|improve this answer

How about when moving (instead of copying):

$ find {origin}/ -maxdepth 1 -name "*" -o -name ".*" -exec mv '{}'  {destination}/ ';'

I think that will move keeping the structure (subdirs) and hidden files or dirs, plus no extra space consumed as with rsync + rm. And if {origin} and {destination} are in the same partition it will be faster.

share|improve this answer

Using tar:

(cd {origin}; tar cf - .)|(cd {destination}; tar xvf -)

Works to get things started when the origin is initially too big for rsync but the deltas are not.

share|improve this answer

You must log in to answer this question.