We have a few thousand small files (2-20KB) to copy between servers in one folder for backup purposes. Currently the process is automated as a cron job and does of SCP copy of each file to the remote server.
These files are are mostly text based so I was thinking
SCP -C ... may work however I'm not sure if there is a way to batch up all these small files (obviously trying to compress each 5KB file is pointless).
I could zip these files up however I would need to unzip them on the other side (need some process running to monitor when the SCP copy had completed and trigger and unzip).
The other option would be to do a ssh and pipe the zip and unzip
So the question is what is the most efficient way to copy many many many small files to a remote server over a WAN while minimizing the number of changes which need to be made on the remote host?