How can I make 'diff -X' ignore specific paths and not file names?

Question

Doing: diff -r -X <ignore-list> <src-dir> <dest-dir>

doesn't seem to make diff ignore entries in <ignore-list> if they are of the form <dir>/<file>.

Entries of the form <file> do however get considered. This is a problem since I might have multiple files named <file> in different sub-directories, some of which I don't want ignored.

There doesn't seem to be much information regarding pattern syntax in the manpage for diff either. From what I can tell, it's just the base-name of a file that is considered by diff (see http://forums.gentoo.org/viewtopic-t-889788-start-0.html if you're interested).

Answer 1

Listing directories should work; e.g., here's what I've used in a script (assuming gnu diff),

diff -r \
   --exclude="*~" \
   --exclude=".svn" \
   --exclude=".git" \
   --exclude="*.zip*" \
   --exclude="*.gz" \
   --exclude="*.tar" \
   ...etc

...which ignores contents of .svn and .git dirs, but also individual files named *.zip/*.gz/etc.

Edit: In order to filter paths of the form dir_a/file1 but still diff files with the same basename, such as dir_b/file1 or dir_a/b/file1, then a list of files to diff would have to be generated (for example, using find), and the file to compare derived from these paths; e.g., given

$ find ONE TWO -type f -print 
ONE/a/1.txt
ONE/a/2.txt
ONE/a/b/2.txt
TWO/a/1.txt
TWO/a/2.txt
TWO/a/b/2.txt

you generate the list of files to compare, excluding for example */a/2.txt but still comparing other files named 2.txt. Just "find" all files except ONE/a/2.txt (a regexp can also be used here, such as .*/a/2.txt)

$ find ONE -type f \( ! -regex 'ONE/a/2.txt' \) \
    -exec bash -c 'diff -q "${1}" "${2/ONE/TWO}"' - {} {} \;  

which in effect ignores ONE/a/2.txt (and TWO/a/2.txt), but still compares the other files named 2.txt:

diff -q ONE/a/1.txt TWO/a/1.txt
diff -q ONE/a/b/2.txt TWO/a/b/2.txt

Edit: Or, more fun with find (additional fun left as an exercise for the reader), select the files or directories to exclude and then diff everything else:

$ find ONE \( -regex 'ONE/a/2.txt' -o -name b  -prune \)  \
    -o -type f -exec bash -c 'echo diff -q "${1}" "${2/ONE/TWO}"' - {} {} \

The above example excludes the specific file "{top}/a/2.txt", any directory named "b", and everything else is diff'd. (Instead of simple "-name b" you could also use "-regex '.*/b'" - note, no trailing "/".)

Answer 2

To exclude directory directory/sub-directory, I use

diff -r <src-dir> <dest-dir> | grep -v directory/sub-directory

However, although it should work for a single exclude, it should not be possible for a long ignore list as you have.

Answer 3

I had the same problem so I created a patch to diff. The patch has yet to be accepted, but you can build your own version of diff with the patch or install on Arch Linux with an AUR package.

Here is the diff patch.

Answer 4

EDIT: Show first the final solution

This one handles spaces in filenames and lets you skip specific paths

The trick to skip a path is to search for " and $NEWER/foo", where $NEWER variable contains the second directory given to the diff command

diff -rqN $OLDER $NEWER | \
sed 's/^Files //;s/ differ$//;' | \
grep -v \
    -e "and $NEWER/data/" \
    -e "and $NEWER/modules_v4/" \
    -e ".*-[bB][aA][kK]-" \
    -e "-[oO][rR][iI][gG][0-9]*" | \
awk -F" and " '{
    tmp=$0;
    s="";
    num=gsub(" and ", "", tmp);
    ocu=(num/2); ocu=(sprintf("%d",ocu+=ocu<0?0:0.9));
    for(i=1;i<=ocu;++i) s = s (s=="" ? "" : FS) $i;
    print substr(s, 1) }' | \
cut -d'/' -f2- | \
xargs -I{} diff -uN $OLDER/{} $NEWER/{} > $ARCH

Now an explanation:

In $OLDER and NEWER variables are stored the directories to compare

1  diff -rqN $OLDER $NEWER | \             Gets a list of changed files 
2  sed 's/^Files //;s/ differ$//;' | \     Removes prefix and suffix added by diff
3  grep -v \                               Remove lines that match (-v)
4      -e "and $NEWER/data/" \             Remove a specific path (directory named data)
5      -e "and $NEWER/modules_v4/" \       Remove a specific path (directory name modules_v4
6      -e ".*-[bB][aA][kK]-" \             Case insensitive file names containing -bak-
7      -e "-[oO][rR][iI][gG][0-9]*" | \    Case insensitive file names containing something like -orig4   (with or without a number at the end)
8  awk -F" and " '{                        AWK to extract file name and handle spaces. 
                                           The diff lines contain something like "filename1 and filename2"
                                           It handles filenames containing " and " without a problem
9      tmp=$0;
10     s="";
11     num=gsub(" and ", "", tmp);         Counts number of times " and " appear
12     ocu=(num/2); ocu=(sprintf("%d",ocu+=ocu<0?0:0.9));   ceil(Middle)
13     for(i=1;i<=ocu;++i) s = s (s=="" ? "" : FS) $i;      finds the position of middle appearance of " and " (without it)
14     print substr(s, 1) }' | \           Substring from beginning to middle appearance of " and "
15 cut -d'/' -f2- | \                      Remove first directory from path
16 xargs -I{} diff -uN $OLDER/{} $NEWER/{} > $ARCH

About the answer given by @wuan

$ diff -rq foo.orig foo | grep -vP 'ignore1/|exclude2/' | awk '{print $2}' | cut -d'/' -f2- | xargs -I{} diff -u foo.orig/{} foo/{}

What this solution does is:

1. Generate a list of file names with changes
2. Filter that list of filenames using grep. In this case shows all files not matching a Perl regular expression (-vP)
3. Take the second filename
4. Removing foo directory at the beginning of the path
5. Execute diff showing contents.

In my opinion is a good idea.

NOTE1: It doesn't handle file names with spaces.

NOTE2: It doesn't handle well cases where there is a file in one directory that does not exist in the other.

I would just modify it a bit to fix this (the "N" flag in the first diff):

$ diff -rqN foo.orig foo | grep -vP 'ignore1/|exclude2/' | awk '{print $2}' | cut -d'/' -f2- | xargs -I{} diff -u foo.orig/{} foo/{}

Or, if you want to include contents of files non existent in the other directory, could use -N option or --unidirectional-new-file option in the second diff:

$ diff -rqN foo.orig foo | grep -vP 'ignore1/|exclude2/' | awk '{print $2}' | cut -d'/' -f2- | xargs -I{} diff -uN foo.orig/{} foo/{}

or

$ diff -rqN foo.orig foo | grep -vP 'ignore1/|exclude2/' | awk '{print $2}' | cut -d'/' -f2- | xargs -I{} diff -u --unidirectional-new-file foo.orig/{} foo/{}

Taken from this web page

If the older directory contains large files that are not in the newer directory, you can make the patch smaller by using the --unidirectional-new-file option instead of -N. This option is like -N except that it inserts the contents only of files that appear in the second directory but not the first (that is, files that were added)

Answer 5

$ diff -rq foo.orig foo | grep -vP 'ignore1/|exclude2/' | awk '{print $2}' | cut -d'/' -f2- | xargs -I{} diff -u foo.orig/{} foo/{}