2

I want to backup some filesystem folders from a client to an external server. My goal is to have a secure backup on the external server even if the client gets compromised/hacked in any way.

I am considering the following techniques (prio in that order):

  • rsync
  • aws S3
  • sFTP
  • HTTPs

AFAIK rsync and sFTP transmit over SSH and therefore I need to put some sort of SSH key on the client. If an attacker gets access to the client, he could also read the SSH key and connect to the external backup server. This is what I want to prevent.

AWS s3 permission model (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html) has a WRITE permissions which "Allows grantee to create, overwrite, and delete any object in the bucket". So a compromised client could delete all of my backups. Not suted in that scenario.

With HTTPs there are of course also credentials stored on the client, but I would create some sort of API on the external system only allowing to POST/Upload data. So an attacker could "only" upload "bulk" data onto the external server, and not e.g. delete my stored backups over SSH. But performing backups via HTTPs sounds like the wrong tool for the job.

Am I missing something obvious?

Any recommendations are highly appreciated.

4
  • 2
    Is this an automated or manual backup? If it's manual, encrypt the SSH/GPG private key. Either way, you'd likely want to configure the server with a chroot for the client to write to and only allow the client to write, not read or execute, creating a user on the server that only has permissions to write to the chroot, with no privileges for anything else. Beyond that, someone with more experience will need to chime in.
    – JW0914
    Commented Oct 7, 2020 at 12:57
  • it is automated (executed daily). I want to store 30 backups in round robin fashion so I thought delete access for the user would also be required. But I guess I can create a local script on the backup server that removes the oldest backup before a new one is generated. Either way can I create a user that only has write access to one directory via ssh and can not perform other operations like su? Im using ubuntu.
    – cpc
    Commented Oct 7, 2020 at 13:25
  • @JW0914 i found this article which describes your approach medium.com/@deltazero/… do you think that is sueted for a production environment?
    – cpc
    Commented Oct 9, 2020 at 12:20
  • 1
    I've never tried that, but it sounds solid and worth a go. To prevent access to the backed up data should the client be compromised, you may want to set up a rsync script with a cron job to monitor the directory within the chroot the backed up data will be sent to, and upon a backup completing, move the data to a permanent location outside of the chroot.
    – JW0914
    Commented Oct 9, 2020 at 12:29

2 Answers 2

3

The way that I handle this is to move control of the data to the backups destination. For the purposes of this description, production is your production (main) machine and backups is the backup server. This does require that you have a backups server rather than just a data repository, though.

  • Remove ssh equivalence from between production and backups unless you protect the keys with a password. (This removes any opportunity for someone or something to get from one to the other.) DO NOT use the same password on both systems!
  • Set up rsyncd on production with a read-everything but read-only configuration protected by username/password. Consider denying any access to ~/.ssh directories - particularly root's
  • Create a regular rsync job on backups to pull data from the production server. I couple this with rsnapshot, and actually to save space I run the entire set of backups on top of S3QL (hung off local disk rather than S3 storage).

This will stop a break-in on production getting to your backups data. It will also stop a break-in on backups overwriting your production system.

The downside of this as described is that rsyncd only delivers data in-the-clear. There are two options to handle this, as I wouldn't recommend clear text data transfer for any trans-Internet communication

  • stunnel, which can be configured to present an SSL socket on behalf of rsyncd
  • a point-to-point VPN tunnel using something such as as Wireguard, OpenVPN or IPsec.

I should point out that although this solution is Linux-centric, I do have rsyncd running on Windows systems, too, successfully backing up user data. The caveat here is that it's not easy to backup data from open files without using a VSS shadow snapshot, and rsyncd doesn't know how to create those.

5
  • nice idea to reverse the roles and pulling the data the other way around. Do you still would NOT recommend this method for trans-Internet communication with stunnelor VPN? Or is it fine then? I do not want to backup whole machines, just some zipped tar.gz archives (few GB). Maybe I could encrypt them prior pulling.
    – cpc
    Commented Oct 9, 2020 at 10:55
  • @capc0 the problem with transferring encrypted data is that authentication (and any other aspect of the rsyncd protocol) is still in the clear Commented Oct 9, 2020 at 11:08
  • A general FYI regarding IPsec: it's not the best solution for data transfer due to its inefficiencies and throughput limitations. Wireguard would be recommended in lieu of, or a tuned OpenVPN config (emphasis on the tuned, as the generic configs found on the internet are generic defaults and not tuned). Wiregaurd has the highest throughput, with OpenVPN second, and IPsec, L2TP, and PPTP always coming in on the bottom end of throughput.
    – JW0914
    Commented Oct 9, 2020 at 12:53
  • @JW0914 thank you for the recommendations on VPNs. (IMO you're a little heavy-handed on the edit; I still don't see any grammatical errors in my earlier revision.) Commented Oct 9, 2020 at 14:18
  • I edit from a professional point of view and as the author, you're able to rollback the edit if you disagree with it (no offense is taken as it's your content)
    – JW0914
    Commented Oct 9, 2020 at 14:46
1

You might consider using BorgBackup in append-only mode.

This involves backing up the client to the server over SSH with key-based authentication. Both the client and the server need to have BorgBackup installed. The backup client will ssh into the backup server and run the borg serve command to handle writing the backup data to the server machine. (This is very similar to how rsync works with an ssh destination.)

You also need to set up an SSH key that the client will use to connect to the server, and configure authorized_keys on the backup server so that the key has a forced command and can only run borg serve --append-only. This mode of borg serve will refuse to delete or prune any data from the backup repository. In order to prune old backups, you need to either run a cron job on the backup server or prune your backups using a second, more-trusted SSH key configured to run borg serve without --append-only.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .