3

I've copied a massive folder from a Windows machine to a Linux machine and due to some files names being too big (and some other errors which I've skipped), some files could not be copied. I'm currently running diff -r between the two folders in order to generate a list of the files that are in the original folder but not the copy. However, so far the only things that it seems to have recognised are missing folders, i.e. it seems to be skipping files. Is there a better way for me to make this comparison? In particularly, I'm worried that Bash simply can't recognise these files with too long file names.

2
  • In general, modern Linux filesystems handle longer file names, longer path names, and deeper folder nesting than Windows' NTFS. Might want to check on the error message that sarted you down this road. I know diff compares the contents of files. Not sure if it works on folders the way you're talking about. SE is for Q&A. A better question would be, "How do I compare two folders to ensure they contain the same sets of files." If you edit your question to reflect what you're trying to do, someone out there probably has a snippet of bash or python code to do just that.
    – Jeter-work
    Jan 23, 2018 at 22:45
  • I just tested. diff -r does catch file changes, even within in subdirectories. You might rather want to use -ur, however. (I find the output much more intelligible that way.)
    – jpaugh
    Jan 23, 2018 at 23:30

4 Answers 4

4

If rsync is a viable option, perhaps the --itemize-changes (-i) and --dry-run options would be of use:

rsync -zaic src_dir/ dest_dir/ --dry-run

-z compresses files during transfer, -a copies in archive mode and -c bases the file comparisons on checksums rather than date modified or size.

-i will list the individual files that are different and --dry-run means that no data will be transferred, just generating a list.

2
  • Is that of use? To my knowledge rsync would require copying the files again.
    – J. Min
    Jan 24, 2018 at 7:37
  • 1
    @J.Min the --dry-run flag doesn't actually copy anything, just shows you what would have happened :)
    – sippybear
    Jan 24, 2018 at 17:48
1

You might do something not entirely unlike:

(cd some/where; ls -lR) > somewhere.txt
(cd else/where; ls -lR) > elsewhere.txt
diff somewhere.txt elsewhere.txt

I haven't tried this, it depends on file metadata (dates etc) being preserved (cp -p ...) and on ls sorting filenames in the same order (which it should).

2
  • Sorry, the some/where notation has got me a bit lost. What would this be in terms of directories?
    – J. Min
    Jan 23, 2018 at 23:09
  • some and else and where are just placeholders for the relative paths to the two parent folders you want to compare (including comparing their subdirectories). e.g. cd /home/jmin/catpics and cd /home/jmin/copy_of_catpics or whatever. Jan 23, 2018 at 23:13
1

diff --recursive (-r) does catch file changes, even within in subdirectories.

You might rather want to use diff --unified --recursive, however. It creates a unified diff, which displays changed lines prefixed with (+) for additon and (-) for removal. Conveniently, it also displays surrounding lines (i.e. context), so that you can figure out what's going on there.

3
  • I'll run that now. Hopefully it'll be done within some reasonable timeframe. Just to be totally sure, that's diff -ur as opposed to diff -r, right? i.e. you're not telling me to run something like diff -r -ur?
    – J. Min
    Jan 24, 2018 at 7:38
  • Back from trying it, after 3 hours I started to get random bursts of lag on my system. By the fourth hour I had no choice but to power off.
    – J. Min
    Jan 24, 2018 at 13:26
  • Yes, I meant diff -u -r. That's crazy! You might need a tool that's optimized for large filesets, then. rsync is probably your best bet.
    – jpaugh
    Jan 30, 2018 at 15:59
1
diff <(cd /first/path/ && find ./ | sort) <(cd /second/path/ && find ./ | sort)

This is similar to this other answer but:

  • I'm using find to generate lists of objects (files, directories); it fits here better than ls because its output contains only paths.
  • sort ensures the relative order of objects is preserved, regardless of in what order each find lists them.
  • The <(…) syntax avoids temporary files in bash.
  • find will be executed only if the corresponding cd succeeds, thanks to the && operator. This will save you from running find in current directory if there's a typo in any path.

Additional notes:

  • Paths returned by find will be relative to directories we cd to. Make sure /first/path/ and /second/path/ correspond to each other.
  • Empty output from diff indicates the two directories are identical; but remember…
  • … the command operates on paths only, it doesn't check if the contents or metadata match.
  • Object names with unusual characters (e.g. with newlines) will break the logic.
1
  • You may simply use sdiff -s as opposed to diff in order to show only the differences from the 2 lists, i.e. the files which are in one dir and not in the other. Jan 25, 2018 at 13:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .