I have a bunch of files (4800 files), that I believe contain many duplicates and triples. Problem is that they don't also have the same names. I already weeded out the easy ones with same names. The files are all in a tight range of size, so that's not helpful.

I thought maybe it might be possible to run a script to check the hashes of all files in a directory, and report duplicate hashes. Anyone know how to achieve such a thing?

Can do on Linux and Windows.


2 Answers 2


Here's a Powershell command that will hash all of your files (in a given directory) and output the result to a CSV file.

Get-FileHash -Algorithm MD5 -Path (Get-ChildItem "\\Path\to\files\*.*" -Recurse) | Export-Csv C:\Temp\hashes.csv

After that, you can open the CSV file in Excel and root out the duplicates.

Note that the "Get-FileHash" cmdlet is only available in Powershell version 4 or higher.

  • Will it check in subdirectories too? Is that what -Recurse does?
    – Dr.Ping
    Jul 14, 2017 at 4:25
  • Yes that is correct. Jul 14, 2017 at 18:11
  • Thanks! Exactly what I needed. Helped me out a ton. Found a few hundred duplicates that other duplicate-file-finders didn't. I changed the algorithm to sha1, to be sure.
    – Dr.Ping
    Jul 15, 2017 at 20:13
  • Once exported into a csv file, Excel's conditional formatting (and then sorting) makes it very easy to find the duplicates
    – Dr.Ping
    Jul 15, 2017 at 20:15
  • It doesn't matter which algorithm you use, as long as you stay consistent. However, using MD5 will be faster overall because it takes less time to calculate each one. Jul 15, 2017 at 22:54

I like Excel But PowerShell has got this covered.

$ChooseDeleteMe = Get-ChildItem -Path "C:\OpenShare" -Recurse -File | Get-FileHash -Algorithm SHA1 | Group-Object -Property Hash | Where-Object Count -gt 1 | foreach {$_.Group | Select-Object Path, Hash} | Sort-Object Hash | Out-GridView -PassThru -Title "Select a Duplicate File OK to Delete (NO PROMPT NO UNDELETE)" | Select-Object -ExpandProperty Path
If ($ChooseDeleteMe.Count -ge 1) {
    Foreach ($DeleteMe in $ChooseDeleteMe) {
        # Remove the WhatIF once outout looks sensible
        Remove-Item -LiteralPath $DeleteMe -WhatIf
} else {
    Write-Warning "No Duplicate Files were Found using the SHA1 Algorithm."

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .