check whether (recursive) directory copy is identical

Question

I sometimes have to copy/sync large amounts of data in a directory structure from one computer to another. Typical programs I use for this purpose are rsync, syncthing, or seafile.

To not exclusively rely on the correctness of the copy/sync program (or the options chosen by me), I usually generate a checksum file (using cfv) in the parent directory and copy/sync it with the data, so I can later check everything is OK on the destination computer. For example, cfv works well to see whether all the files from the origin have safely arrived.

However, an option I miss is to see whether there are any files in the copy which weren't in the original. As far as I can tell, cfv does not have an option to look for such "additional" files. The solution I resort to is to create a new checksum file for the copy and compare it to the original, but that means checksums for every file have to be computed four times (generating & checking, on both computers).

Is there a better solution?

Answer 1

Computing checksums looks like overkill to me if you only want to detect "additional" files. You don't need to check the actual data (file content); you need to check metadata (existing paths).

To get all relative paths inside /synced/dir, run

(cd /synced/dir && find . | sort) > structure.txt

Do it on both sides, then diff the resulting files. Note the situation is symmetrical, so you will detect "additional" as well as "missing" files on any side ("additional" here is equivalent to "missing" there, and vice versa).

To ignore "additional" files on one (or the other) side, filter the diff output with grep '^>' (or grep '^<' respectively).

If the two directories are available (mounted) in a single system, this Bash syntax may be useful:

diff <(cd /original/dir && find . | sort) <(cd /backup/dir && find . | sort) | grep '^>'

This is not totally robust (e.g. newlines in filenames can break the logic), treat my example more like a proof of concept. The point is you detect additional files without reading file contents at all.

Notes:

• sort is needed because the two find-s may return entries in different sequence even if directories are exact copies;
• sole diff can compare directories but this mode is not useful here because it tries to compare contents of corresponding files, this behavior we want to avoid in the first place.

Answer 2

Kamil Maciorowski's answer is very good, but I think his solution can be streamlined in the context outlined in the question, with the following procedure:

1) Create a checksum file on the source. Here's a bash script that does that using cfv:

#!/bin/bash

# create md5 checksum file for all files in the current directory tree

# filename for checksum file
FN="${PWD##*/}.md5"

# create checksum file
cfv -rr -C -L -t md5 -f $FN

It starts from the current directory, descends recursively not following symbolic links, and creates a single checksum file in the current directory.

2) Sync/copy from source to destination.

3) Check the checksum file on the destination (using cfv), and look for additional files using find, sort, and comm:

#!/bin/bash

# test md5 checksum file w.r.t. all files in the current directory tree

# filename for checksum file
FN="${PWD##*/}.md5"

# test checksum file
cfv -T -f $FN

# check whether there are additional files
echo ----------- additional files -----------
CHECK=`tempfile`
sed  's .\{34\}  ' $FN | sort > $CHECK
LOCAL=`tempfile`
find -P -type f -printf '%P\n' | sort > $LOCAL
comm -13 $CHECK $LOCAL

The difference to Kamil Maciorowski's answer is that I don't create a separate file list for the source, but use the filenames in the checksum file, extracted via sed. This assumes that the checksum file is in standard md5sum format: 32 character checksum, space, '*' or ' ' to indicate binary/text mode, filename.