use diff against text files with list of files to find different files by size and/or modified date/time

Question

I need to compare two folders to find files that are either:

• different size and/or modified date/time
• missing from one

I cannot run diff against the two folders in my situation. My plan was to use find on both folders and save the output to two text files and then compare the two text files using diff.

I assume this would work but need to be sure because my source/target directories are huge and if my test shows no difference, or it doesn't find all the differences, I'd have no way of knowing if it worked or not.

If the two folders are exactly the same I assume it would work. But I question what would happen if one folder had a lot more complex sub-directories/files. Will diff be able to understand a folder structure printing output?

For example, I will take an inventory of the folder on one day.

$ find /path/to/folder -exec ls -ld {} \; > inventory-20181101.txt
...

I will modify a bunch of things including add, removing, editing files and adding or removing folders and sub-folders. Then another day I will take another inventory.

$ find /path/to/folder -exec ls -ld {} \; > inventory-20181102.txt
...

Then I will diff the two files.

$ diff inventory-20181101.txt inventory-20181102.txt

I assume this will work if there were no changes or the changes were minor, like just modifying files. But what happens if I add 5 levels of nested folders and then 100 files in it, and remove another top-level folder. Will diff be able to match up the right folders?

Answer 1

To get a reliable overview, you'll need uniform and sortable lists of the files in both directories, and a way to compare these two lists.

As has been pointed out, diff is meant to create readable, semantically sensible overviews of differences between files. This makes it very suitable for comparing plain text or code, but less suitable for comparing lists.
Instead, use comm to find commonalities or differences between two lists.

To generate a "clean" list that only contains the information you need, use the -printf option provided by GNU find. It is more efficient and robust than spawning an ls process for each file, and it can directly output useful information like:

• %Tk File's last modification time in the format specified by k
• %s File's size in bytes
• %p File's name

Putting it all together:

1. List the files in each directory (in a format that only contains the required information) → find … -printf …
2. Sort the lists → sort
3. Find all lines that are not identical between the lists → comm -3: "suppress column 3 (lines that appear in both files)"

 cd dir1 && find . -printf '%T+ %s %p\n' | sort > ../dir1.txt && cd ..
 cd dir2 && find . -printf '%T+ %s %p\n' | sort > ../dir2.txt && cd ..
 comm -3 dir1.txt dir2.txt > differences.txt

One caveat with %T+: the date format will include fractional seconds (2018-11-25+14:58:43.1197033990). If your two directories are stored on different filesystems with different date accuracies, you might have to use a different (manual) date format to exclude the fractional seconds.