I've copied a massive folder from a Windows machine to a Linux machine and due to some files names being too big (and some other errors which I've skipped), some files could not be copied. I'm currently running diff -r between the two folders in order to generate a list of the files that are in the original folder but not the copy. However, so far the only things that it seems to have recognised are missing folders, i.e. it seems to be skipping files. Is there a better way for me to make this comparison? In particularly, I'm worried that Bash simply can't recognise these files with too long file names.
4 Answers
If rsync is a viable option, perhaps the --itemize-changes
(-i) and --dry-run
options would be of use:
rsync -zaic src_dir/ dest_dir/ --dry-run
-z compresses files during transfer, -a copies in archive mode and -c bases the file comparisons on checksums rather than date modified or size.
-i will list the individual files that are different and --dry-run means that no data will be transferred, just generating a list.
-
Is that of use? To my knowledge rsync would require copying the files again.– J. MinJan 24, 2018 at 7:37
-
1@J.Min the --dry-run flag doesn't actually copy anything, just shows you what would have happened :) Jan 24, 2018 at 17:48
You might do something not entirely unlike:
(cd some/where; ls -lR) > somewhere.txt
(cd else/where; ls -lR) > elsewhere.txt
diff somewhere.txt elsewhere.txt
I haven't tried this, it depends on file metadata (dates etc) being preserved (cp -p ...
) and on ls
sorting filenames in the same order (which it should).
-
Sorry, the some/where notation has got me a bit lost. What would this be in terms of directories?– J. MinJan 23, 2018 at 23:09
-
some
andelse
andwhere
are just placeholders for the relative paths to the two parent folders you want to compare (including comparing their subdirectories). e.g.cd /home/jmin/catpics
andcd /home/jmin/copy_of_catpics
or whatever. Jan 23, 2018 at 23:13
diff --recursive
(-r
) does catch file changes, even within in subdirectories.
You might rather want to use diff --unified --recursive
, however. It creates a unified diff, which displays changed lines prefixed with (+) for additon and (-) for removal. Conveniently, it also displays surrounding lines (i.e. context), so that you can figure out what's going on there.
-
I'll run that now. Hopefully it'll be done within some reasonable timeframe. Just to be totally sure, that's diff -ur as opposed to diff -r, right? i.e. you're not telling me to run something like diff -r -ur?– J. MinJan 24, 2018 at 7:38
-
Back from trying it, after 3 hours I started to get random bursts of lag on my system. By the fourth hour I had no choice but to power off.– J. MinJan 24, 2018 at 13:26
-
Yes, I meant
diff -u -r
. That's crazy! You might need a tool that's optimized for large filesets, then. rsync is probably your best bet.– jpaughJan 30, 2018 at 15:59
diff <(cd /first/path/ && find ./ | sort) <(cd /second/path/ && find ./ | sort)
This is similar to this other answer but:
- I'm using
find
to generate lists of objects (files, directories); it fits here better thanls
because its output contains only paths. sort
ensures the relative order of objects is preserved, regardless of in what order eachfind
lists them.- The
<(…)
syntax avoids temporary files inbash
. find
will be executed only if the correspondingcd
succeeds, thanks to the&&
operator. This will save you from runningfind
in current directory if there's a typo in any path.
Additional notes:
- Paths returned by
find
will be relative to directories wecd
to. Make sure/first/path/
and/second/path/
correspond to each other. - Empty output from
diff
indicates the two directories are identical; but remember… - … the command operates on paths only, it doesn't check if the contents or metadata match.
- Object names with unusual characters (e.g. with newlines) will break the logic.
-
You may simply use
sdiff -s
as opposed todiff
in order to show only the differences from the 2 lists, i.e. the files which are in one dir and not in the other. Jan 25, 2018 at 13:38
diff -r
does catch file changes, even within in subdirectories. You might rather want to use-ur
, however. (I find the output much more intelligible that way.)