54

What is the best and simplest way to compare two directory structures without actually comparing the data in files? This works fine:

diff -qr dir1 dir2_

But it's really slow because it's comparing files too. Is there a switch for diff or another simple cli tool to do this?

  • By "directory structure", do you mean just the directory paths, or the paths of both directory and non-directory files? – intuited Jul 22 '10 at 6:26
  • Yes, folders and files. – Jonah Jul 22 '10 at 15:30
  • 1
    In that case you should remove the -type d option from @slartibartfast's answer, or check out my answer. – intuited Jul 22 '10 at 17:52
36

The following (if you substitute the first directory for directory1 and the second for directory2) should do what you're looking for and swiftly:

find directory1 -type d -printf "%P\n" | sort > file1
find directory2 -type d -printf "%P\n" | sort | diff - file1

The fundamental principle is that it prints out all of the directories including subdirectory paths relative to the base directoryN directories.

This could fall down (produce wierd output) if you have carriage returns in some of the directory names but not others.

  • This is no good for me, because if one directory contains a folder with a few thousand files in they are all listed individually, while diff -rq just shows the root directory exists in one, and carries on. – Chris Jefferson Sep 21 '16 at 14:46
  • As pointed out (years ago) by intuited, to answer the OPs question, the -type d should be removed so that files are considered in the comparison as well as directories – user2746401 May 24 '18 at 15:40
  • I understand and respect that reading of the problem statement. That was not my reading at the time. Are you recommending I edit my answer to respond to the updated question? I'm okay doing that if you think it will be helpful to some people, and I'm okay leaving the solution and comment set the way they are now, which seems to be reasonably effective. – Slartibartfast May 25 '18 at 23:12
34
vimdiff <(cd dir1; find . | sort) <(cd dir2; find . | sort)

will give you a nice side-by-side display of the two directory hierarchies with any common sections folded.

  • +1 for nice ui solution without creating additional files – Griddo Nov 19 '13 at 11:52
  • This solution fails randomly. When vim reads (or re-reads) the temporary file descriptor, it is already gone. – Denilson Sá Maia Aug 25 '16 at 17:09
23

I usually use rsync for this task:

rsync -nav --delete DIR1/ DIR2

BE VERY CAREFUL to always use the -n, aka --dry-run, option, or it will synchronize (change the contents of) the directories.

This will compare files based on file modification times and sizes... I think that's what you really want, or at least you don't mind if it does that? I got the sense that you just want it to happen faster, not that you need it to ignore the difference between file contents. If you do want it to not list differing files with identical names, I think the addition of the --ignore-existing option will do that.

Also be aware that not putting a / at the end of DIR1 will cause it to compare the directory DIR1 with the contents of DIR2.

The output ends up being a bit verbose, but it will show you which files/directories differ. Files/directories present in DIR2 and not in DIR1 will be prefaced with the word deleting.

For some situations, @slartibartfast's answer may be more appropriate, though you'll need to remove the -type d option to enable the listing of non-directory files. rsync will be faster if you've got a significant number of files/directories to compare.

  • Excellent answer. In rsync's output it's hard to notice the deleting... text but it's probably one of the better ways to compare files while still maintaining speed. Other' answers here are faster when diffing files isn't required...as in OP's example, but I really like this one. – Joel Mellon Dec 18 '14 at 20:11
  • This is what I was after. I had some files with different sizes in a massive pair of directory trees, and I wanted to know which ones. This achieved that aim in mere seconds. – suprjami Nov 30 '15 at 12:13
  • Maybe it is a good idea to run it with a user that has a read only access. Like sudo -u nobody rsync -nav --delete d1 d2 provided that the flags for 'others' allow reading. – user1182474 Jan 22 '16 at 15:36
  • When running this solution I got "building file list...done\n sent X bytes received Y bytes Z bytes/sec total size is A speedup is B" (where I substituted XYZAB for numbers). Does that mean that everything was identical? Since it didn't mention anything more specific? Thanks in advance – Scott H Jan 4 '18 at 14:17
  • To answer my own question, I experimented adding different files to each, and it appears that no specific files/dirs mentioned in the output means they are all the same. – Scott H Jan 4 '18 at 20:34
18

Similar to the ls answer but if you install tree then you can

tree dir1 > out1
tree dir2 > out2
diff out1 out2
  • 7
    Or to avoid the tmpfiles, diff <( tree dir1 ) <( tree dir2 ) – Joel Mellon Dec 18 '14 at 19:46
  • 1
    I recommend running tree with the i flag, which doesn't print the tree lines (tree -i dir1, etc). If the directory structure is different in one place, the other files that do match may have more or fewer | symbols in the tree output, and diff will catch those lines even if the file paths are identical. – askewchan Dec 10 '15 at 17:31
  • 2
    diff <( tree -i dir1 ) <( tree -i dir2 ) is by far the best answer. I'm tempted to downvote all answers that suggest diff or rsync as the question explicitly says NOT to read the file contents. NOTE: The suggestion of using two pipes requires careful use of spaces between brackets, follow the example exactly. E.g. to compare two 20G volumes after a backup the tree answer took about 5 seconds. The others took 20+ minutes. – Jason Morgan Jan 13 '17 at 12:01
3

I was just looking for solution for this problem. The solution that I liked the most was:

comm <(ls DIR1) <(ls DIR2)

It gives you 3 columns: 1 - files only in DIR1, 2 - files only in DIR2, 3 - files only in DIR3 For more details look at this blog post.

  • Where is DIR3 specified? All I see is DIR1 and DIR2. – Michael Dorst Aug 20 '13 at 23:59
  • I tried it, and (from what I can tell) the output was: all the files only in DIR1 in column 1, all the files only in DIR2 in column 2, and all the files shared by both in column 3. That's sort of useful, but do you know how one might strip out column 3 and leave only the differences? I have a lot of files to sort through, and most of it is identical. I don't need to see what's the same. – Michael Dorst Aug 21 '13 at 0:14
  • 1
    Also, I found that comm <(ls DIR1) <(ls DIR2) did not work recursively. For that I used comm <(ls -R1 DIR1) <(ls -R1 DIR2). ls -R crawls through directories recursively, and ls -1 (note that that is a one, not an L) makes ls print only one filename per line. – Michael Dorst Aug 21 '13 at 0:22
  • @Michael: comm -3 (see man comm). – Zaz Jul 20 '14 at 11:31
2
ls > dir1.txt

ls > dir2.txt

Then just diff the two lists.

  • It seems like the OP wants a heirarchy of paths. This will diff all files in the current directory. It's debatable, but possible, that he just wants directories; he might want filenames rather than the contents of files. – intuited Jul 22 '10 at 6:24
  • @intuited - you're right. I misread it. – MDMarra Jul 22 '10 at 13:16
2

This is optimum solution

diff --brief -r dir1 dir2

--brief switch reports only whether the files differ, not the details of the difference.

  • 1
    The OP already has -q in the question, which is an alias for --brief. This answer doesn't provide any new information. – Michael Dorst Aug 20 '13 at 23:54
  • 1
    OP doesn't want the file contents comparison. But it's really slow because it's comparing files too. – Joel Mellon Dec 18 '14 at 20:06
1

use "diff -qr" to get the different files and then filter out the file comparison with grep in order to only get the filenames that are only in one of the directories.

diff -qr dir1 dir2 | grep -v "Files.*differ" 
-3

I think only rsync is userfull. why?

diff is useful only for structures keeping files and directories. Diff does not give adequate exit codes when we use symlinks. In that situation diff can return 2 exit codes, even if src and dst are identical (times, sizes, names, timestamps, pointing softlinks etc).

dir, the filesystem does not guarantee file ordering, even if directory contents on src and dst are identical. Maybe you should filter the ls output by sorting it. But pure ls displays only node names.

maybe script including diff, cmp, test -X for node types will be usefull, but remember about overload made by many test/cmp runs. The script will be very slow.

As usual, if you want get simple info "dirs is/isn't identical", you should use rsync with the -n (dry) option. If you want to find what is different, use the diff command.

  • I would like to know why minuses? – Znik Mar 10 '16 at 10:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.