Find duplicate hashes

Question

I have a bunch of files (4800 files), that I believe contain many duplicates and triples. Problem is that they don't also have the same names. I already weeded out the easy ones with same names. The files are all in a tight range of size, so that's not helpful.

I thought maybe it might be possible to run a script to check the hashes of all files in a directory, and report duplicate hashes. Anyone know how to achieve such a thing?

Can do on Linux and Windows.

Thanks

Answer 1

Here's a Powershell command that will hash all of your files (in a given directory) and output the result to a CSV file.

Get-FileHash -Algorithm MD5 -Path (Get-ChildItem "\\Path\to\files\*.*" -Recurse) | Export-Csv C:\Temp\hashes.csv

After that, you can open the CSV file in Excel and root out the duplicates.

Note that the "Get-FileHash" cmdlet is only available in Powershell version 4 or higher.

Answer 2

I like Excel But PowerShell has got this covered.

$ChooseDeleteMe = Get-ChildItem -Path "C:\OpenShare" -Recurse -File | Get-FileHash -Algorithm SHA1 | Group-Object -Property Hash | Where-Object Count -gt 1 | foreach {$_.Group | Select-Object Path, Hash} | Sort-Object Hash | Out-GridView -PassThru -Title "Select a Duplicate File OK to Delete (NO PROMPT NO UNDELETE)" | Select-Object -ExpandProperty Path
If ($ChooseDeleteMe.Count -ge 1) {
    Foreach ($DeleteMe in $ChooseDeleteMe) {
        # Remove the WhatIF once outout looks sensible
        Remove-Item -LiteralPath $DeleteMe -WhatIf
    }
} else {
    Write-Warning "No Duplicate Files were Found using the SHA1 Algorithm."
}