Removing Duplicate Files With Powershell

Question

Powershell Has been shipped stock since Windows Vista. I have many clients who download files, forget where they end up redownloading the same exact file. These files if they are duplicates are appended with e a number. ie...file(1) file(2) depending on how many times they redownloaded the file.

How do I remove these duplicates with powershell?

Answer 1

Instead of depending on the filename I would run and store MD5 sum of each file and delete when a duplicate is detected this way so newer version of a file are not discarded.

https://stackoverflow.com/questions/10521061/how-to-get-a-md5-checksum-in-powershell

Answer 2

The implementation depends on how you handle a duplicate.
Here I assumed that the last accessed copy would be the most valuable one to the users. For the purpose of housekeeping, I'd remove "( )" from the Filename of the elected copy. I haven't actually tested it, so please try it with care.

    $files = gci c:\users\*\Downloads -include "* (*).*" -Recurse | Where-Object {$_.name -match "^.*\ \([1-9][0-9]*\)\..*$"}
    foreach ($file in $files) {

        $fileOrgFullName = $file.Directory.FullName + "\" + ($file.BaseName -replace " \([1-9][0-9]*\)$", "") + $file.Extension
        if (test-path $fileOrgFullName) {

            $fileOrg = gi $fileOrgFullName
            if ($file.LastAccessTime -gt $fileOrg.LastAccessTime) {

                cp $file.FullName $fileOrg.FullName
            } 
            rm $file.fullName
        } else {

            ren $file.FullName $fileOrgFullName
        }
    }