secure data backup to remote server: scenario of compromised client

Question

I want to backup some filesystem folders from a client to an external server. My goal is to have a secure backup on the external server even if the client gets compromised/hacked in any way.

I am considering the following techniques (prio in that order):

• rsync
• aws S3
• sFTP
• HTTPs

AFAIK rsync and sFTP transmit over SSH and therefore I need to put some sort of SSH key on the client. If an attacker gets access to the client, he could also read the SSH key and connect to the external backup server. This is what I want to prevent.

AWS s3 permission model (https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html) has a WRITE permissions which "Allows grantee to create, overwrite, and delete any object in the bucket". So a compromised client could delete all of my backups. Not suted in that scenario.

With HTTPs there are of course also credentials stored on the client, but I would create some sort of API on the external system only allowing to POST/Upload data. So an attacker could "only" upload "bulk" data onto the external server, and not e.g. delete my stored backups over SSH. But performing backups via HTTPs sounds like the wrong tool for the job.

Am I missing something obvious?

Any recommendations are highly appreciated.

Answer 1

The way that I handle this is to move control of the data to the backups destination. For the purposes of this description, production is your production (main) machine and backups is the backup server. This does require that you have a backups server rather than just a data repository, though.

• Remove ssh equivalence from between production and backups unless you protect the keys with a password. (This removes any opportunity for someone or something to get from one to the other.) DO NOT use the same password on both systems!
• Set up rsyncd on production with a read-everything but read-only configuration protected by username/password. Consider denying any access to ~/.ssh directories - particularly root's
• Create a regular rsync job on backups to pull data from the production server. I couple this with rsnapshot, and actually to save space I run the entire set of backups on top of S3QL (hung off local disk rather than S3 storage).

This will stop a break-in on production getting to your backups data. It will also stop a break-in on backups overwriting your production system.

The downside of this as described is that rsyncd only delivers data in-the-clear. There are two options to handle this, as I wouldn't recommend clear text data transfer for any trans-Internet communication

• stunnel, which can be configured to present an SSL socket on behalf of rsyncd
• a point-to-point VPN tunnel using something such as as Wireguard, OpenVPN or IPsec.

I should point out that although this solution is Linux-centric, I do have rsyncd running on Windows systems, too, successfully backing up user data. The caveat here is that it's not easy to backup data from open files without using a VSS shadow snapshot, and rsyncd doesn't know how to create those.

Answer 2

You might consider using BorgBackup in append-only mode.

This involves backing up the client to the server over SSH with key-based authentication. Both the client and the server need to have BorgBackup installed. The backup client will ssh into the backup server and run the borg serve command to handle writing the backup data to the server machine. (This is very similar to how rsync works with an ssh destination.)

You also need to set up an SSH key that the client will use to connect to the server, and configure authorized_keys on the backup server so that the key has a forced command and can only run borg serve --append-only. This mode of borg serve will refuse to delete or prune any data from the backup repository. In order to prune old backups, you need to either run a cron job on the backup server or prune your backups using a second, more-trusted SSH key configured to run borg serve without --append-only.