rsync --remove-source-files but only those that match a pattern

Question

Is this possible with rsync?

Transfer everything from src:path/to/dir to dest:/path/to/other/dir and delete some of the source files in src:path/to/dir that match a pattern (or size limit) but keep all other files. I couldn't find a way to limit --remove-source-files with a regexp or size limit.

Update1 (clarification): I'd like all files in src:path/to/dir to be copied to dest:/path/to/other/dir. Once this is done, I'd like to have some files (those that match a regexp or size limit) in src:path/to/dir deleted but don't want to have anything deleted in dest:/path/to/other/dir.

Update2 (more clarification): Unfortunately, I can't simply rsync everything and then manually delete the files matching my regexp from src:. The files to be deleted are continuously created.

So let's say there are N files of the type I'd like to delete after the transfer in src: when rsync starts. By the time rsync finishes there will be N+M such files there. If I now delete them manually, I'll lose the M files that were created while rsync was running.

Hence I'd like to have a solution that guarantees that the only files deleted from src: are those known to be successfully copied over to dest:. I could fetch a file list from dest: after the rsync is complete, and compare that list of files with what I have in src:, and then do the removal manually. But I was wondering if rsync can do this by itself.

Answer 1

Do you want to delete before the transfer or after? That is, should dest: have the deleted files?

I suspect that the easiest way is to do these as separate commands. Use find | xargs rm to delete the desired files (maybe adding an egrep to the pipe if the regex is complex) and rsync to do the copy. The order of execution will be determined by the answer to my question in the first paragraph.

Answer 2

Most likely the "proper" way to do what you want is via find with a function - this lets you do this in an "almost-atomic" way.

I also recommend using mtime - this prevents it from doing files that are still being written to - though there are likely other better ways to avoid that as well.

Also see https://unix.stackexchange.com/questions/50692/executing-user-defined-function-in-a-find-exec-call for related ways to export functions. The example below is bash-specific:

pattern='^match\.this\.pattern.*only$'

rsync_rm() {
 rsync path/to/dir/$1 dest:/path/to/other/dir/.
 echo $1 | egrep $pattern > /dev/null && rm $1
}

export -f rsync_rm

find path/to/dir -mtime +1 -exec bash -c 'rsync_rm "$@"' bash {} +