4

I intend to transfer full and incremental backups of my btrfs subvolumes to a tape archival service. The service exposes FTP and SSH endpoints. If I were allowed to execute arbitrary commands on the SSH endpoint, then I would do the following to perform an incremental backup:

btrfs send -p $LAST_SUBVOLUME $NEXT_SUBVOLUME | compress | encrypt |
    ssh -p $PORT $USER@$ENDPOINT "cat > $SUBVOLUME.$YYYYMMDD.btrfs.bz2.gpg"

I am, however, not allowed to do that:

$ ssh -p $PORT $USER@$ENDPOINT
Last login: Mon Jan  3 01:23:45 2067 from 123.456.789.123

This account is restricted by rssh.
Allowed commands: scp sftp rsync 

If you believe this is in error, please contact your system administrator.

Connection to some.remote.endpoint closed.

So what I thought to do instead was to use the SCP protocol for the transfer. However, my scp binary refuses to transfer a named pipe:

$ scp -P $PORT <(btrfs send -p $LAST_SUBVOLUME $NEXT_SUBVOLUME | compress | encrypt) \
    $USER@$ENDPOINT:$SUBVOLUME.$YYYYMMDD.btrfs.bz2.gpg
/dev/fd/63: not a regular file

The irony is that apparently, scp used to do the right thing once. I suppose not everybody thought transferring named pipes could be sane / useful. EDIT (2017-06-03): This was an incorrect observation. As noted by Kenster, the SCP protocol does not permit sending files of unknown size.

UPDATE (2017-06-04): I also tried transferring the data using the SFTP protocol. Apparently, the FTP protocol permits sending files of unknown size, as evidenced by the support for piping data in the ftp (link) and ncftpput (link, section Description, last paragraph) binaries. I found no such support in the SFTP clients I have tried (sftp, lftp), which might be an indication that SFTP (unlike FTP) does not support sending files of unknown size (contrary to what I thought, SFTP is not just FTP tunelled over SSH; it is a different protocol).

UPDATE (2017-06-05): According to the SFTP protocol version 3 internet draft (link), each application-level packet must specify the payload length before the payload (link, Section 3). However, SFTP supports seeks in the written file (link, Section 6.4) with explicit support for writes beyond the current end of file. Therefore, if should be possible to use a small buffer on the client side and send a file of unknown size in small known-sized chunks:

#!/bin/bash
# <Exchange SSH_FXP_INIT requests.>
# <Send an SSH_FXP_OPEN request.>
CHUNK_SIZE=32768
OFFSET=0
IFS=''; while read -r -N $CHUNK_SIZE CHUNK; do
  ACTUAL_SIZE=`cat <<<"$CHUNK" | head -c -1 | wc -c`
  # <Send an SSH_FXP_WRITE request with payload $CHUNK of size
  # $ACTUAL_SIZE at offset $OFFSET.>
  OFFSET=$(($OFFSET+$ACTUAL_SIZE))
done < <(command)
# <Send an SSH_FXP_CLOSE request.>

However, doing the communication manually over the shell would be quite painful. I am looking for an SFTP client that exposes this kind of functionality.

References

  • I don't see that scp used to handle pipes. The bug report you link shows that it used to hang indefinitely on pipes; now it warns you instead. That sounds like better error reporting, not a change in underlying behaviour. – user4556274 Jun 3 '17 at 11:48
  • Well, I supposed that it had hanged precisely because it would begin to read the named pipe like a regular file. – Witiko Jun 3 '17 at 11:53
3

lftp 4.6.1 and newer should be able to do this: https://github.com/lavv17/lftp/issues/104

Unfortunately, the command suggested in the linked issue does not work, but a slightly-modified one does:

lftp -p $port sftp://$user:$pass@$host -e "put /dev/stdin -o $filename"

Due to a bug, you must provide something in the password field if you use a SSH agent. But any random string will do.

This command reads from /dev/stdin. If you must use a different named pipe, I imagine that should work too - if not, you can always cat named-pipe | lftp ....

I've tested lftp with a >100GB upload without any issues.


rclone also supports SFTP, and should be able to upload from a piped input with the rcat command, due to be released in 1.38 (next version).

| improve this answer | |
  • I am getting a put: /dev/stdin: cannot seek on data source error with the lftp 4.7.4 and 4.8.1 binaries released by the Debian project. I need to check if this is a regression in the source, or if this is a build issue. I will make this the accepted answer after I have managed to reproduce your results. – Witiko Sep 23 '17 at 21:26
  • @Witiko That's interesting - because I tested the current lftp on Debian Stretch. Package version 4.7.4-1. LFTP version 4.7.4, Readline 7.0, GnuTLS 3.5.8, zlib 1.2.8. The full command I ran included a pipe from openssl enc though pv and then into lftp. – Bob Sep 24 '17 at 1:12
  • @Witiko Just as an example, gist.github.com/BobVul/86271ddaae2733d0f888875dd52c4d45 – Bob Sep 24 '17 at 1:55
  • You are right, I misspecified the port, which made lftp emit the above unhelpful error message. Thank you for the advice, it is much appreciated! – Witiko Sep 24 '17 at 9:35
  • @Witiko I think I stumbled on this in the exact same situation you were in (zfs send | compress | encrypt | upload) — it was actually your research that convinced me to go looking down the SFTP path rather than SCP/rsync. So, thank you for a very well-written question (and Kenster for his analysis of SCP/SFTP). Just out of curiosity, is your remote end also C14? – Bob Sep 26 '17 at 1:41
6

SCP isn't well-suited for your purpose. The SCP protocol doesn't support sending an unknown-sized stream of bytes to the remote system to be saved as a file. The SCP protocol message for sending a file requires the size of the file to be sent first, followed by the bytes that make up the file. With a stream of bytes read from a pipe, you typically wouldn't know how many bytes the pipe is going to produce so there's no way to send an SCP protocol message including the correct size.

(The only online descriptions of the SCP protocol that I could find are here and here. Focus on the description of the "C" message.)

The SFTP protocol can be used for this sort of thing. As far as I know, the normal sftp command-line utility doesn't support reading a pipe and storing it as a remote file. But there are SSH/SFTP libraries for most modern programming languages (perl, python, ruby, C#, Java, C, etc). If you know how to use one of these languages, it should be straightforward to write a utility that does what you need.

If you're stuck with shell scripting, it's possible to spoof enough of the SCP protocol to transfer a file. Here's an example:

#!/bin/bash
cmd='cat /etc/group'

size=$($cmd | wc -c)    
{
        echo C0644 $size some-file
        $cmd
        echo -n -e '\000'
} | ssh user@host scp -v -p -t /some/directory

This will create some-file in /some/directory on the remote system with permissions 644. The file contents will be whatever $cmd writes to its standard output. Note that you are running the command twice, with whatever resource consumption and side effects that implies. And the command must output the same number of bytes each time.

| improve this answer | |
  • I would be more than happy to do two passes -- the first one to find out the size and the second one to do the sending via scp; it is not the most elegant solution, but the incremental backups will be all cached in the RAM. Libraries are an option, but I was hoping I would be able to do this from the shell. – Witiko Jun 4 '17 at 8:07
  • Thank you for the research that went into writing that shell script. For me, this is a satisfactory answer to the original question. I am, however, still going to explore the SFTP route, because the speed difference between single-pass and two-pass is going to be noticeable for larger backups. – Witiko Jun 4 '17 at 14:18
2

If you can write perl, you could try using Net::SFTP::Foreign which is able to transfer files from an open file descriptor via SFTP:

#!/usr/bin/perl
# untested!
use Net::SFTP::Foreign;
my $user = ...;
my $host = ...;
my $remote_path = ...;
my $btrfs_cmd = 'btrfs send -p $LAST_SUBVOLUME $NEXT_SUBVOLUME';
my $sftp = Net::SFTP::Foreign->new("$user\@$host", ...);
open my($pipe), "$btrfs_cmd | compress | encrypt |"
    or die $!;
$sftp->put($fh, $remote_path);
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.