How can I quickly compare two directories skipping hard linked files that exist in both directories in linux

Question

I have two directories that I would like to compare. The first directory contains an rsync backup of many thousands of folders and files. The second directory contains hard links to all of the files in the first, plus a few additional folders and files. I would like to be able to compare the directories in such a way that the hardlinked files are skipped or at least quickly decided to be the same file and only those files that are not hardlinks are further compared. Is this something diff can do, or is there a better way?

Answer 1

cmpdir is for this problem (Disclaimer - I wrote it).

It only compares inodes. Files that only exist in the first directory are marked as "Removed", whereas files that only exist in the second directory as "New". It does not compare the file content though.

Answer 2

Probably not fully what you mean, but if you want to skip files that are in both directories, you can use find with the -links option.

find ./ -links 1

shows only files that have one hard link (and cannot therefore be present in both directories).

Answer 3

1. You can show inodes of your files with ls -i -1 (-1 / "one" writes one line per file to output).

2. Save the output for each of the two folders.

3. If the filename of "original" and hardlinked file differ, use cut -f1 to filter the filenames out and retain only the inode numbers.

4. Sort the files for inode number using sort. Now you can use diff to view the differences.

5. Use grep "^<" diffOutput.list to filter for lines starting with < => Inodes occurring only in first folder. grep "^>" analogous for Folder B. Use cut -c2- to delete the first character (< or >) from the output if you need the inode number.

---

If you previously have removed the filenames from the files, you have to search the filenames for a given inode number.

Save the output of ls -l -1 (with filenames) for each folder into one file. Now you can use grep to get the filenames for given inode numbers in a bash loop:

Let InodesFilenamesA.list be the file containing inode numbers and filenames for Folder A

12345 file1inBothFolders
99999 file2OnlyInFolderA
88888 file3OnlyInFolderA
...

Let inodesA.list be a file containing Inodes occurring only in Folder A

99999
88888
....

Now loop over inodesA.list.

cat inodesA.list | while read line; do
    grep "$line" InodesFilenamesA.list
done

You get a list of folder