Here is the situation:
Need to copy aprox 10 million of small files (1k - 50k each) from a single directory, between 2 remote servers over the web. Tried with FTP and SCP but failed since all files are in single directory and somehow freezes the transfer.
Problem is that can not use TAR as on source server SSH is not available, just on destination server where I have full control.
Everyday number of files is increasing by 10-40k so it is getting harder to make final copy, any suggestions will be much appreciated.
Edit: To clarify the situation: source server (where are all files located) is a normal shared hosting server with access to PHP/Mysql and stuff (PHP can execute common linux commands though). Destination server where I want to transfer the files is a full root access (SSH etc) on a VPS instance.
Now, I can tar/zip etc the files but wonder how long it will take to archive all 10/20 million files that I have (small size files). If I do it via PHP at some point a time out will appear or can I send shell exec with run in background mode, or something?
Other option is to pull the files from destination server somehow, in small amounts or ? Any suggestions will be appreciated as I am getting frustrated already. Thanks so much for replys already made.