There are files in two separate folders, one folder has more files than the other but a large number of the files are identical. They take a lot of space, so I want to just keep the files in the second folder that the first folder doesn't have.

How do I do that, without having to go file by file?

  • 2
    Please move your question to serverfault. – mootmoot Jun 27 '17 at 16:05
  • Windows or Linux? – Ivan Jun 27 '17 at 16:54
  • In Linux you can use stat to compare file sizes and modification times: if these are identical, there is a pretty good chance that the contents are also identical, though you cannot be absolutely sure without a full byte-by-byte comparison, such as cmp does. – AFH Jun 27 '17 at 19:43

Two alternatives for Windows are DOS findstr and fc (file compare). I don't know what the exact command would look like, if someone does please comment.


Assuming the filenames are identical in both folders, and you have a Bash shell, run this command from within the second folder:

grep -Ff <(find /path/to/first/folder/*  -printf "%f\n") <(find *  -printf "%f\n") | xargs -0 rm

The second command searches for all files with the same name and rms it. Be careful while deleting. You may want to run the command without | xargs -0 rm first, just to see what files are being deleted.

  • 1
    Piping to rm doesn't work, you need xargs or parallel or similar; the first requires care if any filename contains whitespace, quotes, or backslash, and both fail if any filename contains newline (which is unusual). grep -f may not be optimized for large number of strings (even with -F) so awk or perl hash lookup may be faster (for data that fits in memory, but that's true for grep -f also) – dave_thompson_085 Jun 28 '17 at 2:21
  • Good point @dave_thompson_085. Updated to use xargs. You're right in that this scales quite poorly. – Jedi Jun 28 '17 at 3:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy