I intend to transfer full and incremental backups of my btrfs subvolumes to a tape archival service. The service exposes FTP and SSH endpoints. If I were allowed to execute arbitrary commands on the SSH endpoint, then I would do the following to perform an incremental backup:
btrfs send -p $LAST_SUBVOLUME $NEXT_SUBVOLUME | compress | encrypt | ssh -p $PORT $USER@$ENDPOINT "cat > $SUBVOLUME.$YYYYMMDD.btrfs.bz2.gpg"
I am, however, not allowed to do that:
$ ssh -p $PORT $USER@$ENDPOINT Last login: Mon Jan 3 01:23:45 2067 from 123.456.789.123 This account is restricted by rssh. Allowed commands: scp sftp rsync If you believe this is in error, please contact your system administrator. Connection to some.remote.endpoint closed.
So what I thought to do instead was to use the SCP protocol for the transfer. However, my
scp binary refuses to transfer a named pipe:
$ scp -P $PORT <(btrfs send -p $LAST_SUBVOLUME $NEXT_SUBVOLUME | compress | encrypt) \ $USER@$ENDPOINT:$SUBVOLUME.$YYYYMMDD.btrfs.bz2.gpg /dev/fd/63: not a regular file
The irony is that apparently, EDIT (2017-06-03): This was an incorrect observation. As noted by Kenster, the SCP protocol does not permit sending files of unknown size.
scp used to do the right thing once. I suppose not everybody thought transferring named pipes could be sane / useful.
UPDATE (2017-06-04): I also tried transferring the data using the SFTP protocol. Apparently, the FTP protocol permits sending files of unknown size, as evidenced by the support for piping data in the
ftp (link) and
ncftpput (link, section Description, last paragraph) binaries. I found no such support in the SFTP clients I have tried (
lftp), which might be an indication that SFTP (unlike FTP) does not support sending files of unknown size (contrary to what I thought, SFTP is not just FTP tunelled over SSH; it is a different protocol).
UPDATE (2017-06-05): According to the SFTP protocol version 3 internet draft (link), each application-level packet must specify the payload length before the payload (link, Section 3). However, SFTP supports seeks in the written file (link, Section 6.4) with explicit support for writes beyond the current end of file. Therefore, if should be possible to use a small buffer on the client side and send a file of unknown size in small known-sized chunks:
#!/bin/bash # <Exchange SSH_FXP_INIT requests.> # <Send an SSH_FXP_OPEN request.> CHUNK_SIZE=32768 OFFSET=0 IFS=''; while read -r -N $CHUNK_SIZE CHUNK; do ACTUAL_SIZE=`cat <<<"$CHUNK" | head -c -1 | wc -c` # <Send an SSH_FXP_WRITE request with payload $CHUNK of size # $ACTUAL_SIZE at offset $OFFSET.> OFFSET=$(($OFFSET+$ACTUAL_SIZE)) done < <(command) # <Send an SSH_FXP_CLOSE request.>
However, doing the communication manually over the shell would be quite painful. I am looking for an SFTP client that exposes this kind of functionality.