25

Let's say I copy and paste files from folder A, which includes:

Folder A:

file1.cfg  
file2.txt  
file3.esp  
file4.bsa  

into folder B, which after updating, has:

Folder B:

apples.mp3  
file1.cfg    *
file2.txt    *
file3.esp    *
file4.bsa    *
turtles.jpg

Is there a way to delete all files from folder A that are in folder B (marked with a *)? Besides manually selecting each one and deleting it, or ctrl-Z'ing right after the copy-paste

I would prefer either a windows method or some software that could do this

thanks!

4
  • 4
    How do you know they're the same files content-wise? I can't imagine a scenario where you would want to blindly consider a file to be a duplicate just based on file name alone.
    – rory.ap
    Jun 26, 2016 at 22:10
  • @roryap I think this question was arised because OP copied the files from folder 1 to folder 2, replaced all and now thinks, hmm, this was a mistake, but realizes that the next day, so undo is not possible. But you're right, contentwize you can't know.
    – LPChip
    Jun 27, 2016 at 7:36
  • 13
    Just a dumb question... Why not use "cut" and "paste"?
    – DaMachk
    Jun 27, 2016 at 10:13
  • 1
    @DaMachk if you're working with network drives or removable media, copy->verify->clean-up is a reasonable route. If the files are used by some process, it might be a good idea to test it on a copy (I do this with files for python data analysis in case of bugs in my own code clobbering the input file (for example). It may not be as necessary as it used to be, but old habits and all that. Alternatively the OP could have mis-clicked copy instead of cut,
    – Chris H
    Jun 28, 2016 at 12:27

11 Answers 11

37

There's free software out there called WinMerge. You can use this software to match up duplicates. First, use FileOpen, and choose both directories, with the folder with files you want to keep on the left, and those you do not on the right. Then, go to View, and deselect Show Different Items, Show Left Unique Items, and Show Right Unique Items. This will leave just the identical files left in the list. After that, choose EditSelect All, right-click on any file, and click on DeleteRight. This will delete the duplicates from the right-hand folder.

demo of WinMerge

1
  • The benefit of this method is that it can detect if the files are not content-wise similar, if this is important. WinMerge can compare all factors that matters to one.
    – user477799
    Jun 27, 2016 at 19:59
26

This can be done through the commandline by using the command forfiles

Lets assume you have Folder A located in c:\temp\Folder A, and Folder B located in c:\temp\Folder B

The command would then be:

c:\>forfiles /p "c:\temp\Folder A" /c "cmd /c del c:\temp\Folder B\@file"

After this is done, Folder B will have all files removed that are present in Folder A. Keep in mind, that if folder B has files with the same name, but not the same content, they will still be deleted.

It is possible to extend this to work with folders in subfolders too, but out of fear for this to become unnecessary complicated, I've decided against posting it. It would require the /s and @relpath options (and further testing xD)

0
12

You can use this PowerShell script:

$folderA = 'C:\Users\Ben\test\a\' # Folder to remove cross-folder duplicates from
$folderB = 'C:\Users\Ben\test\b\' # Folder to keep the last remaining copies in
Get-ChildItem $folderB | ForEach-Object {
    $pathInA = $folderA + $_.Name
    If (Test-Path $pathInA) {Remove-Item $pathInA}
}

Hopefully it's fairly self-explanatory. It looks at every item in Folder B, checks whether there's an item with the same name in Folder A, and if so, it removes the Folder A item. Note that the final \ in the folder paths is important.

One-line version:

gci 'C:\Users\Ben\test\b\' | % {del ('C:\Users\Ben\test\a\' + $_.Name) -EA 'SilentlyContinue'}

If you don't care whether you get a deluge of red errors in the console, you can remove the -EA 'SilentlyContinue'.

Save it as a .ps1 file, e.g. dedupe.ps1. Before you can run PowerShell scripts, you'll need to enable their execution:

Set-ExecutionPolicy Unrestricted -Scope CurrentUser

Then you'll be able to invoke it with .\dedupe.ps1 when you're in the folder that contains it.

6

rsync

rsync is a program used to synchronize directory. From the many (really many) options you have there are the self explaining --ignore-non-existing, --remove-source-files and --recursive.

You can do

rsync -avr --ignore-non-existing --recursive --remove-source-files   B/ A -v

if we suppose you have the files in the directory A (4) and B (4+2).

A       B
├── a   ├── a
├── b   ├── b
├── c   ├── c
└── d   ├── d
        ├── e
        └── f     # Before


A       B
├── a   ├── e
├── b   └── f
├── c   
└── d             # After
4

LPChip's answer is the better one.

But because I've started to learn Python, I thought, "Heck, why not write a Python script as answer to this question?"

Install Python and Send2Trash

You'd need to install Python before you can run the script from the command line.

Then install Send2Trash so the deleted files aren't gone irretrievably but end up in the OS's trash:

pip install Send2Trash

Create script

Create a new file with for instance the name DeleteDuplicateInFolderA.py

Copy the following script into the file.

#!/usr/bin/python

import sys
import os
from send2trash import send2trash


class DeleteDuplicateInFolderA(object):
    """Given two paths A and B, the application determines which files are in
       path A which are also in path B and then deletes the duplicates from
       path A.

       If the "dry run" flag is set to 'true', files are deleted. Otherwise
       they are only displayed but not deleted.
    """

    def __init__(self, path_A, path_B, is_dry_run=True):
        self._path_A = path_A
        self._path_B = path_B
        self._is_dry_run = is_dry_run

    def get_filenames_in_folder(self, folder_path):
        only_files = []
        for (dirpath, dirnames, filenames) in os.walk(folder_path):
            only_files.extend(filenames)
        return only_files

    def print_files(sel, heading, files):
        print(heading)
        if len(files) == 0:
            print("   none")
        else:
            for file in files:
                print("   {}".format(file))

    def delete_duplicates_in_folder_A(self):
        only_files_A = self.get_filenames_in_folder(self._path_A)
        only_files_B = self.get_filenames_in_folder(self._path_B)

        files_of_A_that_are_in_B = [file for file in only_files_A if file in only_files_B]

        self.print_files("Files in {}".format(self._path_A), only_files_A)
        self.print_files("Files in {}".format(self._path_B), only_files_B)

        if self._is_dry_run:
            self.print_files("These files would be deleted: ", [os.path.join(self._path_A, file) for file in files_of_A_that_are_in_B])
        else:
            print("Deleting files:")
            for filepath in [os.path.join(self._path_A, file) for file in files_of_A_that_are_in_B]:
                print("   {}".format(filepath))
                # os.remove(filepath)  # Use this line instead of the next if Send2Trash is not installed
                send2trash(filepath)

if __name__ == "__main__":
    if len(sys.argv) == 4:
        is_dry_run_argument = sys.argv[3]
        if not is_dry_run_argument == "--dryrun":
            println("The 3rd argument must be '--dryrun' or nothing.")
        else:
            app = DeleteDuplicateInFolderA(sys.argv[1], sys.argv[2], is_dry_run=True)
    else:
        app = DeleteDuplicateInFolderA(sys.argv[1], sys.argv[2], is_dry_run=False)
    app.delete_duplicates_in_folder_A()

Usage

Dry run mode, which shows you which files would be delete without actually deleting any files:

c:\temp> python .\DeleteDuplicateInFolderA.py c:\temp\test\A c:\temp\test\B --dryrun

File deleting mode, which indeed does delete files, so be careful:

c:\temp> python .\DeleteDuplicateInFolderA.py c:\temp\test\A c:\temp\test\B

Output of dry run mode

Files in C:\temp\A
  1.txt
  2.txt
Files in C:\temp\B
  2.txt
  3.txt
These files would be deleted:
  C:\temp\A\2.txt

Output of file deleting mode

Files in C:\temp\A
  1.txt
  2.txt
Files in C:\temp\B
  2.txt
  3.txt
Deleting files:
  C:\temp\A\2.txt

Unit test

If you want to test the application above, create a file named DeleteDuplicateInFolderATest.py and paste these unittests into it:

import unittest
import os
import shutil
from DeleteDuplicateInFolderA import DeleteDuplicateInFolderA


class DeleteDuplicateInFolderATest(unittest.TestCase):

    def __init__(self, *args, **kwargs):
        super(DeleteDuplicateInFolderATest, self).__init__(*args, **kwargs)
        self._base_directory = r"c:\temp\test"
        self._path_A = self._base_directory + r"\A"
        self._path_B = self._base_directory + r"\B"

    def create_folder_and_create_some_files(self, path, filename_list):
        if os.path.exists(path):
            shutil.rmtree(path)
        os.makedirs(path)
        for filename in filename_list:
            open(os.path.join(path, filename), "w+").close()

    def setUp(self):
        # Create folders and files for testing
        self.create_folder_and_create_some_files(self._path_A, ["1.txt", "2.txt"])
        self.create_folder_and_create_some_files(self._path_B, ["2.txt", "3.txt"])

    def tearDown(self):
        for path in [self._path_A, self._path_B, self._base_directory]:
            if os.path.exists(path):
                shutil.rmtree(path)

    def test_duplicate_file_gets_deleted(self):
        # Arrange
        app = DeleteDuplicateInFolderA(self._path_A, self._path_B, is_dry_run=False)

        # Act
        app.delete_duplicates_in_folder_A()

        # Assert
        self.assertFalse(os.path.isfile(self._path_A + r"\2.txt"), "File 2.txt has not been deleted.")

    def test_duplicate_file_gets_not_deleted_in_mode_dryrun(self):
        # Arrange
        app = DeleteDuplicateInFolderA(self._path_A, self._path_B, is_dry_run=True)

        # Act
        app.delete_duplicates_in_folder_A()

        # Assert
        self.assertTrue(os.path.isfile(self._path_A + r"\2.txt"), "File 2.txt should not have been deleted in mode '--dryrun'")

def main():
    unittest.main()

if __name__ == '__main__':
    main()
8
  • Can you tell me why this script is "ugly as hell"? I just read through it and what you are doing is crystal clear. I'm almost tempted to paste it on CodeReview.SE to learn about what isn't preferred about it. Jun 27, 2016 at 15:45
  • Adding a md5sum to check if the files contents are the same would be a nice option. Also using the OS trash mechanism instead of removing.
    – lolesque
    Jun 27, 2016 at 16:01
  • @user1717828: I've restructured the code, deleted that comment and took your suggestion to post the code on CodeReview.SE.
    – Lernkurve
    Jun 28, 2016 at 21:50
  • @lolesque: Send2Trash part: done. Thank you for the idea!
    – Lernkurve
    Jun 30, 2016 at 8:58
  • 1
    @barlop, I was replying to the original post, not a comment. Jul 17, 2016 at 14:07
1

Using bash

for f in $(ls /path/to/folderB/); do 
    rm -rf /path/to/folderA/$f
done

Sure you could be more more safe by checking if the file is there, or checking if the filename is safe. But assuming you just want to get this done, and don't have any ridiculously named files in folderB - this is a quick and dirty way to get it done. (and you can use the bash emulator that comes with git, if you aren't running Win10 + bash)

1
  • Maybe you need to add a check if you find directories...
    – Hastur
    Jul 12, 2019 at 13:20
1

Any NC-style program, like Total Commander, has a directory difference command that selects files in both tabs that are different from the other tab. Call this command, tab to the larger directory (B), invert selection using * and delete. This has the advantage of not deleting files which may have changed (somehow) and are not the same albeit they agree in name. You can use the same directory diff command to locate these after the deletion.

I guess I'm stuck in the nineties... but I haven't really seen anything more elegant since :-) So far this is the only answer that requires as little as some 5 keystrokes and no scripting / command line whatsoever.

1

Let's say I copy and paste files from folder A into folder B.

Is there a way to delete all files from folder A that are in folder B? Besides manually selecting each one and deleting it, or ctrl-Z'ing right after the copy-paste

Windows Method

If you always need to copy files from one location into another and then afterwards ensure the files which were copied successfully are also deleted from the original source location, then below is a batch script solution that you can use to automate that entire task with just a simple click each run.

  • Be sure to set the SourceDir and the DestDir variables accordingly for your needs.

  • Additionally, in the part of the script below as ("%SourceDir%\*.*") DO you can simply change the *.* value to be more explicit for file names (File A.txt) or file extensions (*.wav) as needed.


@ECHO ON
SET SourceDir=C:\Users\User\Desktop\Source
SET DestDir=C:\Users\User\Desktop\Dest

FOR %%A IN ("%SourceDir%\*.*") DO XCOPY /F /Y "%%~A" "%DestDir%\" && DEL /Q /F "%%~A"
GOTO EOF

Further Resources

0
`

1. Go to Origin Folder: pushd "D:\Dir_Origin"

2. Use one simple for loop to list yours files: for %i in (*)do...

3. Test if exist your file in Target folder: if exist "D:\Dir_Target\%~nxi"

4. If last command return true, delete this current file in loop: del "%~fi"

  • In command-line:
@(pushd "D:\Dir_Origin" && (for %i in (*)do if exist "D:\Dir_Target\%~nxi" del "%~fi")) & popd
  • In bat File:
@(pushd "D:\Dir_Origin" && (for %%i in (*)do if exist "D:\Dir_Target\%%~nxi" del "%%~fi")) & popd
  • One option using dir and operator &&
@(pushd "D:\Dir_Origin" && (for %i in (*)do 2>nul dir /b "D:\Dir_Target\%~nxi" && del "%~fi")) & popd

0

Method without any External Software and Terminal

Just copy the files from folder B to folder A. It will ask for replacing the files which are duplicates. Replace all of them. By replacing, the last modified time of the duplicated (replaced) files changes. Now just sort them using the modified time. Delete the files which are modified currently. You will get the remaining ones.

0

I've adapted Ben N's answer to make it dynamic with multiple folders on both sides (keep and delete).

Keep in mind it will only check for file names, and not the difference between the actual files.

# Pages that were helpful:
# https://docs.microsoft.com/en-us/powershell/scripting/learn/deep-dives/everything-about-arrays?view=powershell-7.2
# https://superuser.com/a/1093799
# https://stackoverflow.com/a/24315319

# DON'T FORGET TO ADD A "\" AT THE END OF THE FILE PATH.
# EXAMPLE: 'C:\Temp\'

# Folders to remove cross-folder duplicates from
$pathsToRemoveFilesFrom = @(
    'C:\Temp\Test1\'
    'C:\Temp\Test2\'
)

# Folders to keep the last remaining copies in
$pathsToKeepThefilesFrom = @(
    'C:\ImportantFiles1\'
    'C:\ImportantFiles2\'
    'C:\ImportantFiles3\'
    'C:\ImportantFiles4\'
)


foreach($pathKeep in $pathsToKeepThefilesFrom) {
    Get-ChildItem $pathKeep -recurse | ForEach-Object {
        foreach($pathDelete in $pathsToRemoveFilesFrom){
            $fullPath = $pathDelete + $_.Name
            If (Test-Path $fullPath) {Remove-Item $fullPath}
        }
    }
}

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .