9

What would be the easiest way to go about rsyncing the n newest files in a directory to a remote server?

1
  • If you want to update, then rsync has an --update switch. You could also use find with head if you want the N newest files, and pipe this into rsync. You may have to use xargs but I don't know for sure.
    – QuickishFM
    Jan 31, 2019 at 12:26

3 Answers 3

6

The easiest way is to run zsh instead of bash.

rsync -a -- /path/to/directory/*(om[1,42]) remote-server:

In the parentheses, om orders files by reverse modification time (i.e. by increasing age), and [1,42] selects the first 42 matches.

If you want only regular files and not directories, add a . after the opening parenthesis. For more possibilities, look under “glob qualifiers” in the zshexpn man page.

If you want to match files in the directory tree rooted at /path/to/directory, rather than just inside that directory, use /path/to/directory/**/*(.om[1,42]). This will send all the files to the same directory on the target, though.

2
  • 1
    FYI, I had to add single quote around '*(om[1,42])' to make it work. I was copying files from remote server to local though.
    – c 2
    Nov 19, 2017 at 18:57
  • 1
    @c2 The quotes cause the wildcards to be expanded on the remote machine. They are needed on a remote path (and will work only if your login shell on the remote machine in zsh), and would not work on a local path. Nov 19, 2017 at 20:28
2

Assuming you want to send files from the current working directory:

rsync `ls -tp | grep -v / | head -n <n>` <destination> <options>

will do the trick. For example:

rsync `ls -tp | grep -v / | head -n 10` user@host:/dest/dir/ --progress --compress

This will give an error if there are no files to be found in the current working directory, or if any of the top files contain spaces or other special characters.

The ` characters around ls -tp | grep -v / | head -n <n> tell bash to run the commands and replaced them with the resulting file list as a space separated list. The -t option tells ls to sort by timestamp, the -p tells it to add a / after directory names and the grep part screens out lines ending / so you don't end up sending directories over. Add -c to the ls options if you want the newest files to be judged by creation time instead of modification time (though note that some programs will remove and replace files instead of updating them so ctime and mtime can be the same even though a file seems to have been around longer).

I'll not claim it is without doubt the easiest way but it would be the way I'd first think of.


(mwm's sub-question-in-a-comment)

what about the other way around? server to local?

That would be a little more complex as I don't think you can have multiple remote file sources on the rsync command line like you can local ones. You could run rsync on the server via over ssh, opening a tunnel back to your local sshd (with the same ssh invocation you run rsync through) to receive the connection, essentially used the same command from the other side.

3
  • Do not use this command unless you know that the file names cannot contain any “special” character (whitespace, \[?*, control characters or non-ASCII characters). Nov 9, 2010 at 0:06
  • what about the other way around? server to local?
    – mwm
    Jan 30, 2019 at 21:53
  • @mwm - more faf, but possible. See note added to answer. Jan 31, 2019 at 12:11
0

I couldn't get the zsh option to work for me, but was able to cook up this:

ls -tr | head -n 10 | rsync --files-from=- user@host:/dest/dir/

Replace 10 with the number of newest files you'd like.

Important: Note that, as with other solutions here, rsync will then reorder these files and send them in the order it wants, typically oldest first :(. I'm still looking for a solution that will have rsync send them with the newest file first. Will need to dig into rsync a little more when I can prioritize it. You can see this by using -vv on the rsync command.

3
  • 1
    ls -tr sorts by oldest first so you should do ls -tr | tail -n 10 to get to 10 newest files.
    – gaoithe
    Nov 9, 2021 at 20:18
  • And when you use --files-from you must still specify source and destination so full command should be ls -tr |tail -n 10 |rsync --files-from=- . user@host:/dest/dir/
    – gaoithe
    Nov 9, 2021 at 20:27
  • And what if you want to do this for all new files and dirs recursively below some directory, and have all files complete paths included in the sync? If my target/destination storage is smaller than the source, and I always only want to just sync/update the newest and/or latest modified files, how do I best accomplish that? I tried many ideas out there, but none seem to actually fulfill this task..
    – Julius
    Feb 21, 2023 at 11:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .