Here is the situation:

Need to copy aprox 10 million of small files (1k - 50k each) from a single directory, between 2 remote servers over the web. Tried with FTP and SCP but failed since all files are in single directory and somehow freezes the transfer.

Problem is that can not use TAR as on source server SSH is not available, just on destination server where I have full control.

Everyday number of files is increasing by 10-40k so it is getting harder to make final copy, any suggestions will be much appreciated.

Thanks, R.

Edit: To clarify the situation: source server (where are all files located) is a normal shared hosting server with access to PHP/Mysql and stuff (PHP can execute common linux commands though). Destination server where I want to transfer the files is a full root access (SSH etc) on a VPS instance.

Now, I can tar/zip etc the files but wonder how long it will take to archive all 10/20 million files that I have (small size files). If I do it via PHP at some point a time out will appear or can I send shell exec with run in background mode, or something?

Other option is to pull the files from destination server somehow, in small amounts or ? Any suggestions will be appreciated as I am getting frustrated already. Thanks so much for replys already made.

  • Is rsync present? That might work. Though if you don't have tar, I kinda doubt you have rsync. – Zoredache Apr 4 '13 at 18:47
  • You could use tar (cpio, ... pick your preferred weapon) with netcat if that was available ... if security is a concern you're just out of luck. – tink Apr 4 '13 at 18:49
  • Zip? CPIO? Bzip? You have to have some archiver on the source box? Some way to combine multiple files into one file? – allquixotic Apr 4 '13 at 18:49
  • What kind of access you have in the source server? If its a web server with PHP support and FTP access, you build a PHP file that create a ZIP or TAR for you. – Havenard Apr 4 '13 at 18:51
  • Wget might be the answer here. – crayzeewulf Apr 4 '13 at 19:22

You said, "Everyday number of files are increasing." If you can stop it, recursively pull all files from FTP server using wget:

wget -m ftp://username:password@ip.of.old.host

If you can't stop new file addition until you move fully to new server, use curlftpfs (on new server) to mount FTP host as local directory. Then, use cp with -u and r flags. You can use this in multiple sessions after operation interuption (-u takes care of this).
After mounting FTP host, you can also use rsync.

  • Very interesting idea...I will investigate further this solution and decide what to do and post results back here. I can not stop since this means to take website down but I will use second option. My only concern is how much it will take since mounting via FTP and trying to copy all files at once requires to SCAN all files in directory and for sure it will lock/freeze server...and cause (maybe) performance issues on second (don't know here please confirm). Kind regards,R. – BookMaster Apr 5 '13 at 17:45

Depending on how the files are named, you could try to tar/zip chunks and then try SCP or FTP. You didn't specify the type of file but if they are logs or other text then you should get reasonable compression. Use wildcards to archive all the files beginning with a, foo, bar123 etc.
For example:
tar -czvf chunk01.tar.gz a*
tar -czvf chunk02.tar.gz b*
tar -czvf chunk03.tar.gz c*
tar -czvf chunk03.tar.gz d*
tar -czvf chunk03.tar.gz e*

  • Thanks for your solution. Files are HTML and should compress quite good. All files have standard naming where variable part is in the middle of the file so I guess I can use wildcard on them. Wonder how much time it will take to compress a chunk of 100.000 files, minutes/hours or how...I know it depends on servers HDD/CPU etc but raw estimate ? – BookMaster Apr 5 '13 at 17:04

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.