Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a directory containing around 2.8 lacs of files. I want to move them to another directory.

If I use cp or mv then I get an error 'argument list too long'.

If I write a script like

for file in ls *; do
   cp {source} to {destination} 
done

then, because of the ls command, its performance degrades.

How can I do this?

share|improve this question

migrated from stackoverflow.com Feb 11 '10 at 8:59

This question came from our site for professional and enthusiast programmers.

    
"Lac" as in million? What command did you try first, presumably mv * newDir/ ? –  Paolo Feb 10 '10 at 14:13
    
What is the total size of all files? Maybe first tar these files? –  skwllsp Feb 10 '10 at 14:16
    
See this question. –  Nick Presta Feb 10 '10 at 14:20
4  
technically a lac is one hundred thousand.that and crore are hardly used outside the indian subcontinent though –  Journeyman Geek Feb 16 '10 at 9:39
2  
en.wikipedia.org/wiki/Lakh –  endolith Jul 2 '10 at 1:25

7 Answers 7

Use rsync:

$ rsync -a {source}/ {destination}/

e.g.

$ rsync -a /some/path/to/src/ /other/path/to/dest/

(note the trailing /s)

share|improve this answer
    
bash: /usr/bin/rsync: Argument list too long Sorry Paul!!!!! –  Ritesh Sharma Feb 10 '10 at 15:13
    
@Ritesh -- I'm guessing you specified some files or * as part of {source} - it should just be a directory, e.g. rsync -a /some/path/src/ /other/path/to/ -- note the trailing /s. –  Paul Feb 10 '10 at 15:38
    
Yes Paul. I gave the path of the directory. but it didn't work! –  Ritesh Sharma Feb 11 '10 at 4:08
    
@Ritesh - that doesn't seem possible - can you copy and paste the actual rsync command and resulting error message(s) from your terminal ? –  Paul R Feb 12 '10 at 8:41

You don't need the ls, you can simply use

for file in *; do
    cp $file /your/dest
done

or you can do something like:

echo * | xargs -i cp {} /your/dest
share|improve this answer
    
The first solution won't work coz of performance issue but I should try the second one. I'll but after some time. Thanks. –  Ritesh Sharma Feb 10 '10 at 14:24
#!/bin/bash
d=$(date +%Y%m%d%H%m%s)
cd /path
tar zcvf "/destination/bakup_${d}.tar.gz" mydirectory_for_transer
share|improve this answer
    
I think I should go for this. But one question still ticks in my mind and that is performance? –  Ritesh Sharma Feb 10 '10 at 14:29
1  
i do not have a million files to test, so i can't answer for you about performance. you have to test out yourself on a development server. –  user31894 Feb 11 '10 at 3:21

The problem is the * expansion which fails because there are too many entries. a safe workaround which always works is instead of using ls * is simply using ls (thus not using the * expansion (which is done by your shell, not by the ls command)

for i in ls; do
   cp $i $dest
done 

Or offcourse the ls | xargs or the find solution presented elsewhere.

share|improve this answer

Assuming you want to move the files within the same filesystem, you could just rename the directory containing your lacs and be done with it.

share|improve this answer

I like rsync for this, or:

find dir1 -type f -exec cp {} dir2 \;
share|improve this answer

How about when moving (instead of copying):

$ find {origin}/ -maxdepth 1 -name "*" -o -name ".*" -exec mv '{}'  {destination}/ ';'

I think that will move keeping the structure (subdirs) and hidden files or dirs, plus no extra space consumed as with rsync + rm. And if {origin} and {destination} are in the same partition it will be faster.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.