Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a directory containing around 2.8 lacs of files. I want to move them to another directory.

If I use cp or mv then I get an error 'argument list too long'.

If I write a script like

for file in ls *; do
   cp {source} to {destination} 
done

then, because of the ls command, its performance degrades.

How can I do this?

share|improve this question

migrated from stackoverflow.com Feb 11 '10 at 8:59

This question came from our site for professional and enthusiast programmers.

    
"Lac" as in million? What command did you try first, presumably mv * newDir/ ? –  Paolo Feb 10 '10 at 14:13
    
What is the total size of all files? Maybe first tar these files? –  skwllsp Feb 10 '10 at 14:16
    
See this question. –  Nick Presta Feb 10 '10 at 14:20
4  
technically a lac is one hundred thousand.that and crore are hardly used outside the indian subcontinent though –  Journeyman Geek Feb 16 '10 at 9:39
2  
en.wikipedia.org/wiki/Lakh –  endolith Jul 2 '10 at 1:25
add comment

7 Answers

Use rsync:

$ rsync -a {source}/ {destination}/

e.g.

$ rsync -a /some/path/to/src/ /other/path/to/dest/

(note the trailing /s)

share|improve this answer
    
bash: /usr/bin/rsync: Argument list too long Sorry Paul!!!!! –  Ritesh Sharma Feb 10 '10 at 15:13
    
@Ritesh -- I'm guessing you specified some files or * as part of {source} - it should just be a directory, e.g. rsync -a /some/path/src/ /other/path/to/ -- note the trailing /s. –  Paul Feb 10 '10 at 15:38
    
Yes Paul. I gave the path of the directory. but it didn't work! –  Ritesh Sharma Feb 11 '10 at 4:08
    
@Ritesh - that doesn't seem possible - can you copy and paste the actual rsync command and resulting error message(s) from your terminal ? –  Paul R Feb 12 '10 at 8:41
add comment

You don't need the ls, you can simply use

for file in *; do
    cp $file /your/dest
done

or you can do something like:

echo * | xargs -i cp {} /your/dest
share|improve this answer
    
The first solution won't work coz of performance issue but I should try the second one. I'll but after some time. Thanks. –  Ritesh Sharma Feb 10 '10 at 14:24
add comment
#!/bin/bash
d=$(date +%Y%m%d%H%m%s)
cd /path
tar zcvf "/destination/bakup_${d}.tar.gz" mydirectory_for_transer
share|improve this answer
    
I think I should go for this. But one question still ticks in my mind and that is performance? –  Ritesh Sharma Feb 10 '10 at 14:29
1  
i do not have a million files to test, so i can't answer for you about performance. you have to test out yourself on a development server. –  user31894 Feb 11 '10 at 3:21
add comment

The problem is the * expansion which fails because there are too many entries. a safe workaround which always works is instead of using ls * is simply using ls (thus not using the * expansion (which is done by your shell, not by the ls command)

for i in ls; do
   cp $i $dest
done 

Or offcourse the ls | xargs or the find solution presented elsewhere.

share|improve this answer
add comment

Assuming you want to move the files within the same filesystem, you could just rename the directory containing your lacs and be done with it.

share|improve this answer
add comment

I like rsync for this, or:

find dir1 -type f -exec cp {} dir2 \;
share|improve this answer
add comment

How about when moving (instead of copying):

$ find {origin}/ -maxdepth 1 -name "*" -o -name ".*" -exec mv '{}'  {destination}/ ';'

I think that will move keeping the structure (subdirs) and hidden files or dirs, plus no extra space consumed as with rsync + rm. And if {origin} and {destination} are in the same partition it will be faster.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.