8

I've got a couple of disks which have a large set of files which are mostly the same. However, in a few cases there are files on one disk which differ from files on the other disk. There are also a lot of files where the files are identical on both drives, but the timestamps differ.

For my purposes I need to find just the files which are actually different. If I run:

rsync --dry-run -HPrlt

it find not only the files that are different, but also the files that differ only by timestamp, leaving me with extra work to determine if these are false positives or not.

I also thought to try:

rsync -c --dry-run -HPrlt

But this command takes a lot longer. In fact, the former command ran in a few seconds (presumably because the directory structure was already in cache from a previous rsync) whereas the latter command is still running. I suspect that this is because rsync is relying entirely on the checksum to determine what files need transferring, instead of something slightly more intelligent like only using a checksum if the timestamps differ.

How can I quickly see only the files that actually differ?

Note: This is not a duplicate of How to print files that would have been changed using rsync? because as the highest rated comment to the highest rated answer points out, --dry-run will show files that are identical if their timestamps differ.

8
  • In "But this command takes a lot faster." I think you meant "... is a lot slower".
    – bitinerant
    Commented Feb 6, 2021 at 18:25
  • @bitinerant thanks, I have fixed it
    – Michael
    Commented Feb 6, 2021 at 18:27
  • So, do I understand correctly that want rsync --dry-run and then, of the files that are listed, rsync -c --dry-run? In other words, compare everything by timestamp first, and then checksum files who's timestamps don't match?
    – bitinerant
    Commented Feb 6, 2021 at 18:31
  • @bitinerant i don't know... if i don't use "-c" it shows me files that differ only by timestamp. if I use "-c" it takes too long for the command to run.
    – Michael
    Commented Feb 6, 2021 at 18:31
  • Try rsync --dry-run -HPrl (i.e. without the --times, if you don't want files listed that only differ by timestamp).
    – bitinerant
    Commented Feb 6, 2021 at 18:33

3 Answers 3

6

rsync -HPrl --itemize-changes --dry-run source/ destination/ | grep -Fv "f..T......"| grep -Fv "d..T......"| cut -d " " -f 2

*Do Not Miss Trailing Slash For The Source Directory.

'--itemize-changes' outputs a change-summary for all updates. When combined with --dry-run and greping out files/dirs for which only timestamp update is required, gives the required output (quickly).

8
  • This doesn't work. It still shows ALL the files.
    – shivams
    Commented Jul 28, 2023 at 12:52
  • The command is now mainly comparing file sizes, to identify differing files. Could you check size of a file which you know is same but is wrongly shown in the output? If the sizes are different, try checking their checksum.
    – mittal
    Commented Jul 29, 2023 at 5:45
  • Check this: superuser.com/questions/1448465/…
    – mittal
    Commented Jul 29, 2023 at 5:45
  • 1
    Add a trailing slash for the source directory, as indicated in the answer (added a warning too). Else it's trying to put dir1 inside dir2. It works.
    – mittal
    Commented Jul 31, 2023 at 5:38
  • 1
    Aah. Your solution actually works! It was a matter of trailing slashes. Too bad that the bounty expired though, and I saw it late. My 100 points were gone and I couldn't award them to you!
    – shivams
    Commented Jul 31, 2023 at 11:54
1

It's possible to run rsync in two stages :

  1. Generate a list of all files differing in size or timestamp (which might wrongly include some files that are identical)
  2. rsync using this list and with the checksum-compare option to find the real differences.

This answer is based on the post
Reuse rsync --dry-run output to speed up the actual transfer later on.

To use the file list generated during a dry run as an include file, requires removing the extra lines at the top and bottom of the dry-run output.

Example output:

sending incremental file list
[LIST OF FILES]

sent 226 bytes  received 34 bytes  520.00 bytes/sec
total size is 648,373,274  speedup is 2,493,743.36 (DRY RUN)

To remove the superfluous lines and leave only the files-list:

rsync --dry-run -avz source/ destination/ | head --lines=-3 | tail --lines=+3 > include.txt

For rsync to use this file (add additional options as desired):

rsync -c --include-from=include.txt --exclude=* source/ destination/

EDIT: I have reproduced the problem according to the poster's gist and can add to the description that whenever --dry-run is specified, then all the files are marked for sync, no matter which combination of parameters is used.

I think that the problem is actually with --dry-run, perhaps because it's checking too many metadata attributes. Seems perhaps like a bug.

6
  • Hi @harrymc, thanks for your response. However, this does not fully answer the key bounty issue. The key bounty issue is this: "when running --dry-run, rsync seems to print ALL the files". Why is this happening? I later found this Unix.SE post which discusses the issue but the solutions provided there still don't work: unix.stackexchange.com/questions/341413/…
    – shivams
    Commented Jul 27, 2023 at 11:33
  • @shivams: Could you add more information about source and target disks and how do the files differ in such a way that they are all different?
    – harrymc
    Commented Jul 27, 2023 at 11:53
  • I am on Ubuntu 22.04 using ext4 partition. Here is a gist to reproduce the issue: gist.github.com/shivams/4371e8ee68d39d9b1b7f0c2ccccd725e
    – shivams
    Commented Jul 27, 2023 at 12:24
  • Try to use cp -p in the gist.
    – harrymc
    Commented Jul 27, 2023 at 12:34
  • Yes, used that (and updated the gist). Still the same issue. Can you reproduce it at your end?
    – shivams
    Commented Jul 27, 2023 at 12:38
0

This could be an XY problem: you have an issue to solve, but are asking how to solve it with rsync.

The OP question asks about rsync, but its possible that a timestamp difference will always show as "different". I'm not sure theres a "-c but ignore timestamp" option. No matter what tool you decide to use, you must read the entire file to verify its content.

Here is a possible alternative (non-rsync) solution:

Hash the trees and find the differences. This will produce a list of files that differ. By "differ" I mean any of:

  • the content changed
  • the file exists on one side but not the other
cd /tree1
find -type f -print0 | sort -z | xargs -0 md5sum > /tmp/tree1.log

cd /tree2
find -type f -print0 | sort -z | xargs -0 md5sum > /tmp/tree2.log

diff -uw /tmp/tree1.log /tmp/tree2.log | grep '^[+-]' | awk '{print $2}' | sort -u

(For the md5sum naysayers: I know md5 is broken in the sense of of finding a second preimage attack...but the OP (probably) isn't looking for something cryptographically critical and md5 is faster than sha256.)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .