I am trying to run rsync to copy some files recursively down a path based on their file name pattern, case insensitive. This is what I have done to run rsync:

$ rsync -avvz --include ='*/' --include='.*[Nn][Aa][Mm][E].*' --exclude='*' ./a/ ./b/

Nothing gets copied, the debug output shows:

[sender] hiding file 1Name.txt because of pattern *
[sender] hiding file 1.txt because of pattern *
[sender] hiding file 2.txt because of pattern *
[sender] hiding file Name1.txt because of pattern *
[sender] hiding directory test1 because of pattern *
[sender] hiding file NaMe.txt because of pattern *

I have tried using: --include='*[Nn][Aa][Mm][E]*'and other combinations but it still doesn't go.

Any ideas on how to use regex to include some files?

migrated from stackoverflow.com Jan 15 '13 at 14:35

This question came from our site for professional and enthusiast programmers.

  • 4
    Why are you using the --exclude='*'? – Lorenzo L Jan 14 '13 at 10:50
  • 1
    so it excludes everything that is not part of the include. – user1957413 Jan 14 '13 at 14:40
  • 'hiding file 1Name.txt because of pattern ' this indicates:-" does that --exclude rule need to be in the command ?" or If you want to exclude some files then why a "". – Akshay Patil Jan 14 '13 at 19:29

rsync doesn't speak regex. You can enlist find and grep, though it gets a little arcane. To find the target files:

find a/ |
grep -i 'name'

But they're all prefixed with "a/" - which makes sense, but what we want to end up with is a list of include patterns acceptable to rsync, and as the "a/" prefix doesn't work for rsync I'll remove it with cut:

find . |
grep -i 'name' |
cut -d / -f 2-

There's still a problem - we'll still miss files in subdirectories, because rsync doesn't search directories in the exclude list. I'm going to use awk to add the subdirectories of any matching files to the list of include patterns:

find a/ |
grep -i 'name' |
cut -d / -f 2- |
awk -F/ '{print; while(/\//) {sub("/[^/]*$", ""); print}}'

All that's left is to send the list to rsync - we can use the argument --include-from=- to provide a list of patterns to rsync on standard input. So, altogether:

find a/ |
grep -i 'name' |
cut -d / -f 2- |
awk -F/ '{print; while(/\//) {sub("/[^/]*$", ""); print}}' |
rsync -avvz --include-from=- --exclude='*' ./a/ ./b/

Note that the source directory 'a' is referred to via two different paths - "a/" and "./a/". This is subtle but important. To make things more consistent I'm going to make one final change, and always refer to the source directory as "./a/". However, this means the cut command has to change as there will be an extra "./" on the front of the results from find:

find ./a/ |
grep -i 'name' |
cut -d / -f 3- |
awk -F/ '{print; while(/\//) {sub("/[^/]*$", ""); print}}' |
rsync -avvz --include-from=- --exclude='*' ./a/ ./b/
  • Tried to run it, ran into issues with the cut command. Seems that -t is that a valid switch. – user1957413 Jan 15 '13 at 6:42
  • edit: i meant -t is not a valid switch – user1957413 Jan 15 '13 at 11:18
  • sorry, should be -d. i started off using sed and then changed to cut because i thought it was clearer, but forgot to edit my commands :S – sqweek Jan 15 '13 at 13:31
  • Follow up: Tried to edit the scrip to take arguments ($1 = path_to_search, $2 as the pattern for egrep )as i am matching filename + mix of extensions. That parts works fine, i got the expected list, however rsync fails to copy. It seems to only work with the single name character directory as in the example (a) my guess is that the cut command has to be modified to cut characters based on the parent / or source dir? Kinda lost of how to do that : – user1957413 Jan 18 '13 at 10:15
  • Ah yeah, you are quite right. It should work on a directory name of any length, but will fail as soon as you refer to a directory outside the current directory (because there will be a different number of slashes in the prefix portion). To fix that, probably easiest to use sed instead of cut, like: sed "s#^$1/*##" buuuut that will break on paths that contain a #. To fix that we have to quote the incoming directory name: prefix=$(echo "$1" | sed 's#/#\\/#g') and then sed "s/^$prefix\\/*//" The subleties of bash quoting are a bit of a nightmare ;) – sqweek Jan 19 '13 at 7:49

I would suggest to use the filter option of rsync. For your example just type:

rsync -vam -f'+ *[Nn][Aa][Mm][E]*' -f'+ */' -f'- *' a b

the first filter rule tells rsync what patterns to include. The second rule is needed to tell rsync to inspect all directories on its traversal. To prevent empty dirs from inclusion they are excluded explicitly by -m option. The last filter rule tells rsync to dispose all remaining patterns that still didn't match so far.

  • Sweet. This worked as well. I was getting the folder a inside of b, that got fixed by using a/ b/ as the source and destination. Thanks! – user1957413 Jan 16 '13 at 6:04
  • Use -f'+ *[Nn][Aa][Mm][E]**' (two stars at the end) to include the contents of all directories with a specific name. – phobic Nov 11 '16 at 13:51

If you use ZSH then you can use the (#i) flag to turn off case sensitivity. Example:

$ touch NAME
$ ls (#i)*name*

ZSH also supports exclusions, which are specified just like the regular path but they have an initial ~

$ touch aa ab ac
$ ls *~*c
aa ab

You can chain exclusions:

$ ls *~*c~*b

Finally you can specify what kind of file you want returned (directory, file, etc). This is done with (/) for directory and (.) for file.

$ touch file
$ mkdir dir
$ ls *(.)

Based on all this, I would do that command as:

rsync -avvz *(/) (#i)*name* ./a/ ./b/

(I don't see a need for an exclusion with these selectors)


@sqweek's answer above is awesome, though I suspect he has a bug in his awk script for generating parent directories, as it gives me e.g.:

$ echo a/b/c/d | awk -F/ '{print; while(/\//) {sub("/[^/]*", ""); print}}'

I was able to fix it by using gensub instead:

$ echo a/b/c/d | awk -F/ '{print; while(/\//) { $0=gensub("(.*)/[^/]*", "\\1", "g"); print}}'

So, his full solution, with the awk bit changed, would be:

find ./a/ |
grep -i 'name' |
cut -d / -f 3- |
awk -F/ '{print; while(/\//) { $0=gensub("(.*)/[^/]*", "\\1", "g"); print}}' |
rsync -avvz --include-from=- --exclude='*' ./a/ ./b/
  • Thanks. Edited my answer with the equivalent fix of anchoring the regex to the end of line (sub("/[^/]*$")). – sqweek May 21 '17 at 0:05

Tried with a C# script since is the language i have the most experience with. I am able to create the list of files that i want to include, but someone rsync is still tell me take a hike. It creates the folders, but it ignores the files. Here is what is what i got..

First the content of the directory:

~/mono$ ls -l
total 24
drwxr-xr-x 5 me me 4096 Jan 15 00:36 a
drwxr-xr-x 2 me me 4096 Jan 15 00:36 b
drwxr-xr-x 3 me me 4096 Jan 14 00:31 bin
-rw-r--r-- 1 me me 3566 Jan 15 00:31 test.cs
-rwxr-xr-x 1 me me 4096 Jan 15 00:31 test.exe
-rwxr--r-- 1 me me  114 Jan 14 22:40 test.sh

Then the output of the C# script:

~/mono$ mono test.exe


And the debug output:

~/mono$ mono test.exe | rsync -avvvz --include='*/' --include-from=- --exclude='*' ./a/ ./b/
[client] add_rule(+ */)
[client] parse_filter_file(-,20,3)
[client] add_rule(+ /a/myfile/myfileseries.pdf)
[client] add_rule(+ /a/myfile2/testfile.pdf)
[client] add_rule(- *)
sending incremental file list
[sender] make_file(.,*,0)
[sender] hiding file 1Name.txt because of pattern *
[sender] showing directory myfile2 because of pattern */
[sender] make_file(myfile2,*,2)
[sender] hiding file 1.txt because of pattern *
[sender] hiding file 2.txt because of pattern *
[sender] hiding file Name1.txt because of pattern *
[sender] showing directory test1 because of pattern */
[sender] make_file(test1,*,2)
[sender] hiding file NaMe.txt because of pattern *
[sender] showing directory myfile because of pattern */
[sender] make_file(myfile,*,2)
send_file_list done
send_files starting
[sender] hiding file myfile/myfileseries.pdf because of pattern *
[sender] hiding file myfile2/testfile.pdf because of pattern *
[sender] hiding file test1/test.txt because of pattern *

[EDIT] This only works locally. For remote paths, the directory structure has to be created first.

More simple than the accepted answer; Use --file-from, which includes parent directories automatically and printf the file path with %P

find /tmp/source -wholename '*[Nn][Aa][Mm][E]*' -printf '%P\n' | rsync -vzrm --exclude='*/' --files-from=- /tmp/source/ /tmp/target/

So you only have to use find and rsync.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.