SSH into remote machine, run find command, then rsync the results of the find back to the local machine?

Question

Basically the title.

Here is what I have had success with so far:

A) this successfully finds the file I need and displays it in the terminal of my local machine:

ssh remoteuser@remotemachine find /home/remoteuser/exampledirectory/ -type f -name 'examplefile*' | xargs echo

B) this successfully rsyncs the needed file from the remote machine to my local machine:

rsync -avzhe ssh --progress remoteuser@remotemachine:/home/remoteuser/exampledirectory/examplefile123.tar.gz /home/localuser/examplelocaldirectory

These two commands work fine when I am there to interact with it; i.e. i can see the output of the find and then manually run the second command, but what I am struggling with is tying these two concepts together as a single command so it can be run all at once (so it can be automated via a cronjob). In other words, I need to ssh into the remote machine, find the needed file, and rysync the results of the find back to my local machine, all in one go.

I have private-key no-password ssh access from my local machine into the remote machine. I do not have private-key no-password ssh access from the remote machine into my local machine, but probably could set it up if necessary. I am wondering if there is a way to do this entirely from the local machine though.

Answer 1

Finding files

this successfully finds the file

No. Your first command only luckily finds the file. There are few problems with it. You were lucky it gave you a useful output.

In your manual approach | xargs echo is almost a no-op. Almost, because it may introduce ambiguity. This may not be a problem if there is just one match and its name is not troublesome in this context. I won't explain what "troublesome in this context" means because even if | xargs echo was completely fine, your original first command would still work without it anyway.

Your ssh command is somewhat flawed on its own. This is your command without the unneeded xargs:

ssh remoteuser@remotemachine find /home/remoteuser/exampledirectory/ -type f -name 'examplefile*'

It's good examplefile* is quoted but it's not enough. The quotes prevent the local shell from globbing. Then ssh recognizes the first operand that looks like a command to run on the remote side and builds an acutal command from it and every operand that follows. In your case these operands are find, /home/remoteuser/exampledirectory/, -type, f, -name and examplefile*. The last one is without quotes because the quotes was "used" by the local shell. In effect the remote command built by ssh and run by a shell on the remote side is

find /home/remoteuser/exampledirectory/ -type f -name examplefile*

and this does trigger globbing on the remote side. In some circumstances this may give you an error or unexpected results (see this).

One interesting scenario. If examplefile* is expanded by globbing on the remote side to examplefile123.tar.gz (because such file happens to be in the current working directory and it's the only match), the -name primary will get this exact string, not the pattern you typed, and the whole search will be limited to this exact filename. The command I present below is not flawed like this, -name gets the pattern for sure. Depending on what files there are on the remote side, it may be my command will find more files than yours.

This is the right command:

ssh remoteuser@remotemachine 'find /home/remoteuser/exampledirectory/ -type f -name "examplefile*"'

Single-quotes embrace the entire code destined to the remote side. Double-quotes will survive and prevent globbing on the remote side.

You can store the result in a file on the local side like this:

ssh remoteuser@remotemachine 'find /home/remoteuser/exampledirectory/ -type f -name "examplefile*"' > filelist

---

Passing result to rsync

There is this question: How to rsync only a specific list of files? The best answer advises --files-from. This is what man 1 rsync says:

--files-from=FILE
Using this option allows you to specify the exact list of files to transfer (as read from the specified FILE or - for standard input). […]

Please read the entire relevant fragment of the manual, especially what it says about implied -R:

The --relative (-R) option is implied, which preserves the path information that is specified for each item in the file (use --no-relative or --no-R if you want to turn that off).

After you run the last command provided above, you will have filelist to use with --files-from. An example command:

rsync -avzhe ssh --progress --files-from=filelist --no-relative remoteuser@remotemachine:/ /home/localuser/examplelocaldirectory

On the local side paths will be like

/home/localuser/examplelocaldirectory/examplefile123.tar.gz

Without --no-relative you would get (on the local side) paths like

/home/localuser/examplelocaldirectory/home/remoteuser/exampledirectory/examplefile123.tar.gz

Note I used / as the source (remote) path. This is because you gave an absolute path to find, so all paths in filelist are absolute; but rsync will treat them relative to the directory you specify. Therefore the tool should start in (remote) / to get to the right objects.

But if you used find ./exampledirectory/ … then rsync should be given remoteuser@remotemachine:. (or just remoteuser@remotemachine:) because all paths in filelist would be relative to remoteuser's home directory. In such case without --no-relative the local path would be created like

/home/localuser/examplelocaldirectory/exampledirectory/examplefile123.tar.gz

And if the remote command was

cd exampledirectory && find . …

then all paths in filelist would be relative to remoteuser's ~/exampledirectory. rsync should use remoteuser@remotemachine:exampledirectory. Without --no-relative the local path would be

/home/localuser/examplelocaldirectory/examplefile123.tar.gz

which looks like the one where --no-relative was used. The option still makes a difference for subdirectories. If the (remote) file was …/exampledirectory/foo/examplefile123.tar.gz then you would get

/home/localuser/examplelocaldirectory/examplefile123.tar.gz

or

/home/localuser/examplelocaldirectory/foo/examplefile123.tar.gz

depending on whether or not you used --no-relative.

---

Hints and improvements

• --files-from=- will make rsync read its stdin. You want a single command, so this will be very useful:

  ssh … 'find …' | rsync … --files-from=- …
  

• You were talking about "the needed file", one file; but in general you may get multiple paths from find. rsync will have no problem in processing them all. There are few ways to limit what you get:

  1. More restrictive find. E.g. find … -print -quit (if -quit is supported) to get just one result (but it may not be the one you expect).
  2. Filtering on the remote side. E.g. ssh … 'find … | sort -R | head -n 1' to get just one result, randomly chosen.
  3. Filtering on the local side. E.g. ssh … 'find …' | sort -R | head -n 1.

  In general filtering on the remote side will reduce the amount of data your ssh pulls. But if the remote machine is slow/limited and the filter stresses the CPU or requires lot of memory, you may want to pull more data and parse it on the local side.

• In Linux filenames can contain newline characters. If this happens in your case then a single path will look like multiple entries in the list. rsync will try to retrieve nonexistent (or wrong) files (or existing directories!). If find on the remote side supports -print0 (it may or may not), use it as the final action, along with -0 option of rsync. The entries will be null-terminated. The commands will look like this:

  ssh … 'find … -print0' > filelist
  rsync -0 …
  

  Note if you want to process/filter the list before passing to rsync, all tools you use must treat null characters as delimiters.

You can combine these hints. Example:

ssh … 'find … -print0 | head -z -n 1' | rsync -0 … --files-from=- …