I've copied a massive folder from a Windows machine to a Linux machine and due to some files names being too big (and some other errors which I've skipped), some files could not be copied. I'm currently running diff -r between the two folders in order to generate a list of the files that are in the original folder but not the copy. However, so far the only things that it seems to have recognised are missing folders, i.e. it seems to be skipping files. Is there a better way for me to make this comparison? In particularly, I'm worried that Bash simply can't recognise these files with too long file names.

  • In general, modern Linux filesystems handle longer file names, longer path names, and deeper folder nesting than Windows' NTFS. Might want to check on the error message that sarted you down this road. I know diff compares the contents of files. Not sure if it works on folders the way you're talking about. SE is for Q&A. A better question would be, "How do I compare two folders to ensure they contain the same sets of files." If you edit your question to reflect what you're trying to do, someone out there probably has a snippet of bash or python code to do just that. – Xalorous Jan 23 '18 at 22:45
  • I just tested. diff -r does catch file changes, even within in subdirectories. You might rather want to use -ur, however. (I find the output much more intelligible that way.) – jpaugh Jan 23 '18 at 23:30

If rsync is a viable option, perhaps the --itemize-changes (-i) and --dry-run options would be of use:

rsync -zaic src_dir/ dest_dir/ --dry-run

-z compresses files during transfer, -a copies in archive mode and -c bases the file comparisons on checksums rather than date modified or size.

-i will list the individual files that are different and --dry-run means that no data will be transferred, just generating a list.

| improve this answer | |
  • Is that of use? To my knowledge rsync would require copying the files again. – J. Min Jan 24 '18 at 7:37
  • 1
    @J.Min the --dry-run flag doesn't actually copy anything, just shows you what would have happened :) – sippybear Jan 24 '18 at 17:48

You might do something not entirely unlike:

(cd some/where; ls -lR) > somewhere.txt
(cd else/where; ls -lR) > elsewhere.txt
diff somewhere.txt elsewhere.txt

I haven't tried this, it depends on file metadata (dates etc) being preserved (cp -p ...) and on ls sorting filenames in the same order (which it should).

| improve this answer | |
  • Sorry, the some/where notation has got me a bit lost. What would this be in terms of directories? – J. Min Jan 23 '18 at 23:09
  • some and else and where are just placeholders for the relative paths to the two parent folders you want to compare (including comparing their subdirectories). e.g. cd /home/jmin/catpics and cd /home/jmin/copy_of_catpics or whatever. – RedGrittyBrick Jan 23 '18 at 23:13

diff --recursive (-r) does catch file changes, even within in subdirectories.

You might rather want to use diff --unified --recursive, however. It creates a unified diff, which displays changed lines prefixed with (+) for additon and (-) for removal. Conveniently, it also displays surrounding lines (i.e. context), so that you can figure out what's going on there.

| improve this answer | |
  • I'll run that now. Hopefully it'll be done within some reasonable timeframe. Just to be totally sure, that's diff -ur as opposed to diff -r, right? i.e. you're not telling me to run something like diff -r -ur? – J. Min Jan 24 '18 at 7:38
  • Back from trying it, after 3 hours I started to get random bursts of lag on my system. By the fourth hour I had no choice but to power off. – J. Min Jan 24 '18 at 13:26
  • Yes, I meant diff -u -r. That's crazy! You might need a tool that's optimized for large filesets, then. rsync is probably your best bet. – jpaugh Jan 30 '18 at 15:59
diff <(cd /first/path/ && find ./ | sort) <(cd /second/path/ && find ./ | sort)

This is similar to this other answer but:

  • I'm using find to generate lists of objects (files, directories); it fits here better than ls because its output contains only paths.
  • sort ensures the relative order of objects is preserved, regardless of in what order each find lists them.
  • The <(…) syntax avoids temporary files in bash.
  • find will be executed only if the corresponding cd succeeds, thanks to the && operator. This will save you from running find in current directory if there's a typo in any path.

Additional notes:

  • Paths returned by find will be relative to directories we cd to. Make sure /first/path/ and /second/path/ correspond to each other.
  • Empty output from diff indicates the two directories are identical; but remember…
  • … the command operates on paths only, it doesn't check if the contents or metadata match.
  • Object names with unusual characters (e.g. with newlines) will break the logic.
| improve this answer | |
  • You may simply use sdiff -s as opposed to diff in order to show only the differences from the 2 lists, i.e. the files which are in one dir and not in the other. – AnythingIsFine Jan 25 '18 at 13:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.