Here is an alternative solution:
All of the examples above suggesting ssh+cat assume that "cat" is available on the destination system.
In my case the system (Hetzner backup) had a very restrictive set of tools offering sftp, but not a complete shell. So using ssh+cat was not possible. I came up with a solution that uses undocumented "scp -t" flag. The complete script may be found below.
#!/bin/bash
function join_e
{
for word in $*; do
echo -n "--exclude=$word "
done
}
CDATE=`date +%Y%m%d%H%M%S`
# Make password available to all programs that are started by this shell.
export OPASS=YourSecretPasswrodForOpenSslEncryption
#-----------------------------------------------
# Directory and file inclusion list
ILIST=(
var/lib
)
# Directory and file exclusion list
ELIST=(
var/lib/postgresql
)
# 1. tar: combine all files into a single tar archive
# a. Store files and directories in ILIST only.
# b. Exclude files and directories from ELIST.
# 2. xz: compress as much as you can utilizing 8 threads (-T8)
# 3. openssl: encrypt contents using a password stored in OPASS local environment variable
# 4. cat: concatenate stream with SCP control message, which has to be sent before data
# a. C0600 - create a file with 600 permissions
# b. 107374182400 - maximum file size
# Must be higher or equal to the actual file size.
# Since we are dealing with STDIN, we have to make an educated guess.
# I've set this value to 100x times my backups are.
# c. stdin - dummy filename (unused)
# 5. ssh: connect to the server
# a. call SCP in stdin (-t) mode.
# b. specify destination filename
nice -n 19 bash -c \
"\
tar $(join_e ${ELIST[@]}) -cpf - -C / ${ILIST[*]} \
| xz -c9e -T8 \
| openssl enc -aes-256-cbc -pass env:OPASS \
| cat <(echo 'C0600 107374182400 stdin') - \
| ssh [email protected] "\'"scp -t backup-${CDATE}.tar.xz.enc"\'"\
"
Update 2019.05.08:
As per request, below is a much simpler and shorter version.
#!/bin/sh
# WORKS ON LARGE FILES ONLY
cat filename.ext \
| cat <(echo 'C0600 107374182400 stdin') - \
| ssh [email protected] 'scp -t filename.ext'
Update 2020.01.18:
Hetzner SFTP connections time out pretty fast. To extend timeouts use the following ssh options:
#!/bin/sh
# ...
| ssh -o TCPKeepAlive=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=3 [email protected] 'scp -t filename.ext'