66

Note that I can't first store the file locally -- it's too big.

This (obnoxious) page (scroll all the way to the bottom) seems to give an answer but I'm having trouble disentangling the part that's specific to tape drives:

http://webcache.googleusercontent.com/search?q=cache:lhmh960w2KQJ:www.experts-exchange.com/OS/Unix/SCO_Unix/Q_24249634.html+scp+redirect&cd=3&hl=en&ct=clnk&gl=us

To make this more concrete, here's how you would think it might work:

On local machine:

% echo "pretend this string is a huge amt of data" | scp - remote.com:big.txt

(That's using the convention -- which scp does not in fact support -- of substituting a dash for the source file to tell it to get it from stdin instead.)

1
  • Can you post the url of your google result? Experts Exchange only shows the answer at the bottom if your referrer is google...
    – Jon
    Jun 21, 2010 at 19:52

10 Answers 10

95

You can pipe into ssh and run a remote command. In this case, the remote command is cat > big.txt which will copy stdin into the big.txt file.

echo "Lots of data" | ssh [email protected] 'cat > big.txt'

It's easy and straightforward, as long as you can use ssh to connect to the remote end.

You can also use nc (NetCat) to transfer the data. On the receiving machine (e.g., host.example.com):

nc -l 1234 > big.txt

This will set up nc to listen to port 1234 and copy anything sent to that port to the big.txt file. Then, on the sending machine:

echo "Lots of data" | nc host.example.com 1234

This command will tell nc on the sending side to connect to port 1234 on the receiver and copy the data from stdin across the network.

However, the nc solution has a few downsides:

  • There's no authentication; anyone could connect to port 1234 and send data to the file.
  • The data is not encrypted, as it would be with ssh.
  • If either machine is behind a firewall, the chosen port would have to be open to allow the connection through and routed properly, especially on the receiving end.
  • Both ends have to be set up independently and simultaneously. With the ssh solution, you can initiate the transfer from just one of the endpoints.
6
  • If it's any consolation, I'm inclined to mark yours accepted since it explains nicely what's actually going on. (If you wanted to really clinch it, you could include the FIFO pipe and netcat solutions with some guidance on why you might prefer one or the other! :)
    – dreeves
    Jun 21, 2010 at 20:06
  • Done. Netcat is a handy utility. :) Jun 21, 2010 at 20:37
  • As in the other comment, if you are piping through tar, you can use a process substitution: tar -cvzf >(ssh destination 'cat > file') huge_directory_tree
    – Taywee
    Jul 6, 2013 at 0:49
  • 1
    SSH is indeed the way to go. Compared to nc it also offers encryption and compression of your data by default and more importantly: error-detection. I had situations where I used nc with a faulty network driver and corrupted data got transmitted undetected. SSH will fail in that situation, because it can't decrypt/decompress faulty data.
    – jlh
    Jul 18, 2016 at 11:10
  • As this is the top google result I thought I'd add you can also use gzip compression for better transfer speed and to save storage space if you are using an intermediate machine for the transfer: echo "Lots of data" | gzip -9 | ssh [email protected] 'cat > big.txt.gz'
    – Andy C
    Jun 4, 2020 at 8:16
20

Using ssh:

echo "pretend this is a huge amt of data" | ssh [email protected] 'cat > big.txt'
2
  • Ah, beautiful, thank you! Any reason to prefer this or FIFO pipe solution?
    – dreeves
    Jun 21, 2010 at 19:58
  • I think the mknod approach accomplishes the task in exactly the same way except for the named pipe.
    – bpf
    Jun 21, 2010 at 20:03
5

Use nc (Net Cat), which doesn't need to save the file locally.

2
  • Ah, thanks! Want to include the equivalent of my "echo .. | scp .." example? And any reasons you know of to prefer this to the other answers?
    – dreeves
    Jun 21, 2010 at 20:09
  • 3
    I strongly recommend against using nc for this. I once dumped a raw disk image from one machine to another only to find out much later that my network driver was faulty and transferred faulty bits. Use scp, ssh or anything else that will tell you when there's a transmission error.
    – jlh
    Oct 10, 2016 at 19:07
4

Here is an alternative solution:

All of the examples above suggesting ssh+cat assume that "cat" is available on the destination system.

In my case the system (Hetzner backup) had a very restrictive set of tools offering sftp, but not a complete shell. So using ssh+cat was not possible. I came up with a solution that uses undocumented "scp -t" flag. The complete script may be found below.

#!/bin/bash

function join_e
{
  for word in $*; do
    echo -n "--exclude=$word "
  done
}

CDATE=`date +%Y%m%d%H%M%S`

# Make password available to all programs that are started by this shell.
export OPASS=YourSecretPasswrodForOpenSslEncryption

#-----------------------------------------------

# Directory and file inclusion list
ILIST=(
  var/lib
)

# Directory and file exclusion list
ELIST=(
  var/lib/postgresql
)

# 1. tar: combine all files into a single tar archive
#      a. Store files and directories in ILIST only.
#      b. Exclude files and directories from ELIST.
# 2. xz: compress as much as you can utilizing 8 threads (-T8)
# 3. openssl: encrypt contents using a password stored in OPASS local environment variable
# 4. cat: concatenate stream with SCP control message, which has to be sent before data
#      a. C0600 - create a file with 600 permissions
#      b. 107374182400 - maximum file size
#         Must be higher or equal to the actual file size.
#         Since we are dealing with STDIN, we have to make an educated guess.
#         I've set this value to 100x times my backups are.
#      c. stdin - dummy filename (unused)
# 5. ssh: connect to the server
#      a. call SCP in stdin (-t) mode.
#      b. specify destination filename

nice -n 19 bash -c \
   "\
   tar $(join_e ${ELIST[@]}) -cpf - -C / ${ILIST[*]} \
   | xz -c9e -T8 \
   | openssl enc -aes-256-cbc -pass env:OPASS \
   | cat <(echo 'C0600 107374182400 stdin') - \
   | ssh [email protected] "\'"scp -t backup-${CDATE}.tar.xz.enc"\'"\
   "

Update 2019.05.08:

As per request, below is a much simpler and shorter version.

#!/bin/sh

# WORKS ON LARGE FILES ONLY

cat filename.ext \
| cat <(echo 'C0600 107374182400 stdin') - \
| ssh [email protected] 'scp -t filename.ext'

Update 2020.01.18:

Hetzner SFTP connections time out pretty fast. To extend timeouts use the following ssh options:

#!/bin/sh

# ...
| ssh -o TCPKeepAlive=yes -o ServerAliveInterval=30 -o ServerAliveCountMax=3 [email protected] 'scp -t filename.ext'
1
  • Can you improve this by removing all the unnecessary things and just pare this down to the bare minimum example of how to use scp -t? Right now you have a complete script that's highly custom/localized to your environment. A good thing for the Hetzner wiki, but not for Super User where most people are just looking for how to pipe input through scp. May 7, 2019 at 2:58
2

Use a FIFO pipe:

mknod mypipe p
scp mypipe destination &
ls > mypipe
4
  • I couldn't get this to work on a Linux system. scp complained that mypipe was not a regular file. Jun 21, 2010 at 20:05
  • 1
    Didn't work on a Mac, either, for the same reason. (I had to use mkfifo to create the pipe, though.) Jun 21, 2010 at 20:24
  • 1
    Didn't work here either. But if it does work for you, and you have bash or zsh, you could better achieve this with a process substitution, like for this example: scp <(ls) destination
    – Taywee
    Jul 6, 2013 at 0:48
  • See here: superuser.com/a/1519565/340651 Jan 23, 2020 at 14:27
2

Thanx Denis Scherbakov!

When I tried your script on the Hetzner cloud I got

debug1: Sending command: scp -v -t backup-20180420120524.tar.xz.enc
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
Transferred: sent 4168, received 2968 bytes, in 0.0 seconds
Bytes per second: sent 346786.6, received 246944.0

But only a file with no content got created. As the actual content is already encrypted with openssl, we actually don't need scp. The linux builtin ftp also has great piping capabilities. So here's my (still quite manual) solution:

#!/bin/bash

function join_e
{
  for word in $*; do
    echo -n "--exclude=$word "
  done
}


# Directory and file inclusion list
ILIST=(
  /home
)

# Directory and file exclusion list
ELIST=(
  var/lib/postgresql
)



export OPASS=fileencryptionpassword

nice -n 19 bash -c \
   "\
   tar $(join_e ${ELIST[@]}) -cpvf - -C / ${ILIST[*]} \
   | xz -c9e -T8 \
   | openssl enc -aes-256-cbc -pass env:OPASS \
   "

# decrypt with:
# cat backup.tar.xz.enc | openssl  aes-256-cbc -d  -pass env:OPASS | xz -dc | tar xv

# invocation procedure for ftp:
# $ ftp -np
# ftp> open storage.com
# ftp> user  storageuser storagepass
# ftp> put "| bash ~/backup.sh" backup.tar.xz.enc
1

1) Create fifo pipe and cat on Remote machine:

Remote# mknod /tmp/mypipe p # mkfifo /tmp/mypipe
Remote$ cat /tmp/mypipe > big.txt& exit

2) Send data to remote fifo pipe from Local machine with scp:

Local$ echo "the huge amt of data string" | scp /dev/stdin user@Remote:/tmp/mypipe
1

I had a similar problem with MacOS/Zsh. Based on Ray Butterworth's answer here and this from Adaephon elsewhere on SuperUser this worked for me:

scp =(echo "Hi There!") remote_server:/Users/me/new_file.txt

Just make sure to give a filename in the second argument to scp, otherwise you'll get an unexpected name on the remote server.

0

You can use curl for this job if you can't use cat at the remote server for example. Using sftp protocol you can copy anything you send to its stdin via ssh. You can use ssh key instead of password to authenticate. I didn't try size limitation, but it should work like other solutions and is authenticated and syntax is clean.

Just don't try to switch curl to scp protocol, because it won't work. scp-not-working

echo "pretend this string is a huge amt of data" | curl -k -T - -u user:password sftp://remote.host/big.txt
0
echo "Lots of data" | ssh [email protected] 'tee big.txt'

With sudo:

echo "Lots of data" | ssh [email protected] 'sudo -S tee big.txt'

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .