I am already familiar and use rsync, scp transfers, tar, 7z etc however none of which solve a backup I intend to do rather regularly.

I have a remote server with 80% used space on 100GB SSD that I would like to backup from. Server has 10,000+ small files so a basic tarball or some sort of intermediate compression (like rsync -z) will help.

I have a server at home that I would like to backup to but it uses Dynamic IP.

The trouble is to use rsync -z or scp and tar (via pipe), it has to be done on remote server but that server can't see my home server since its on Dynamic IP.

I can therefore only execute commands on my home server like scp user@remoteserver:/files/to/backup /home/user/backups but this can't use compression so the huge number of files takes very long time to transfer.

Is there a faster way?

P.S. I don't need to store file permissions etc that tarball does

Edit: Because of limited space on remote server, I can't make a tarball there and pull on home server. Therefore, it has to be realtime.

migrated from serverfault.com Dec 1 '18 at 9:30

This question came from our site for system and network administrators.


Why can't use initiate the rsync from your backup target? There is no inherent restriction in rsync that would prevent this. Running

rsync -avz user@remote.example.com:/data /backupdir 

on the backup target (home server) should do just that.

  • I agree, I do exactly this kind of thing to backup my own server. Initiating the copy on the server opens a serious can of worms because your home server has to accept connections from the internet, it can of course be done but making sure it is (and remains) watertigh isn't so easy. – xenoid Dec 1 '18 at 16:27

There are lots of options The easiest might be to just add the -C command to SSH or so which enables compression for the link.

A more sophisticated ,(and faster, be sure of lots f small files and latency) solution would be to drop scp and use a pipe to compress data on the fly. Something like tar -cz /path/to/backup | SSH user@remote "cd /path; tar -xz" will compress and decompress files using gzip compression. Of-course, there is nothing to stop you using a different command after ssh if, for example you just wanted to store the compressed file you could use "cat > /path/to/file.tar.gz" instead.

It's unclear to me why you are talking about dynamic IPs here. If this poses a problem, you can use dynamic DNS or set up a VPN with the server on the static or initiate a pull from the dynamic IP rather then push from the static.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy