# Compare Two Files and Remove the Duplicates

What is the easiest way to compare 2 files and deleting the duplicates found on both files. I'm aware you can do this easily on a linux box with certain commands, but how would I go about achieving this on a windows computer? I would of thought there would be a way to do this on Notepad++ but I can't seem to find one.

The other answers on here don't support a user on windows or don't work.

• Does this answer your question? Notepad++ Compare two files and remove – Moab Jan 21 at 18:12
• The methods mentioned in this post don't work because their fixes either require a linux box or simply don't work. – holograms Jan 21 at 18:14
• @holograms - If you are using Windows 10 version 1903+, you have access to WSL 2, so those Linux solutions are now native Windows solutions. You could also use PowerShell or any number of third-party software solutions. – Ramhound Jan 21 at 18:16
• @Ramhound sadly i'm on windows 8 – holograms Jan 21 at 18:26
• There are dozens of third-party software applications that will find and list any potential duplicate files on a system. If the suggested solutions in the duplicate are not acceptable you should use one of those. – Ramhound Jan 21 at 19:09

I'm aware you can do this easily on a Linux box with certain commands, but how would I go about achieving this on a Windows computer?

GNUWin32

If you're willing to install additional third-party software, the GNUWin (GNUWin32) project offers Windows ports of a number of these utilities (commands).

The utilities themselves are offered in packages with either a single utility or several utilities bundled in each. Each package can be installed separately 1, with the exception of certain packages where the included utilities are dependent on others.

The basic package you will most likely want to install is the CoreUtils package but there are other packages available as well (depending on what utilities you may wish to use).

For what it's worth, I've included some simple example batch files using GNUWin32 utilities which may help you in another section of this answer.

GNUWin32 Alternatives

Setting aside the Windows Subsystem for Linux (WSL) in Windows 10, there are other possible options for using these types of utilities on Windows, such as Git for Windows and Cygwin.

These versions often have the advantage of being more up-to-date than their GNUWin32 counterparts, but arguably may be lacking in ease of installation and use.

I am personally not aware of any Notepad++ plugins or features that fully support what you're trying to accomplish. However, Notepad++ can run batch scripts, commands and executables directly via the Notepad++ Run... menu ( F5 ) or the NppExec plugin and thus can be used (to some degree) in concert with e.g. GNUWin32 utilities.

With that said, I am not sure of a good mechanism for feeding separate open files as a group to anything listed above. One partial workaround is to combine the two files into one, sort them, save the file and then run a variation of the "Single File Output" example batch file given later in this answer against that saved file.

Notepad++ does support more low-level interactions with the editor via the Python Script for Notepad++ plugin (based around a self-contained version of Python 2.7). Unfortunately, my experience with that plugin is limited so I am uncertain how it might be able to help you towards the desired functionality (if at all).

If you want to experiment with the Python Script for Notepad++ plugin yourself, you can install it under the Plugins → Plugins Admin... interface in Notepad++.

GNUWin32 Batch Examples

The batch (.bat) file examples given below leverage GNUWin32 utilities and are intended show some rudimentary approaches to (hopefully) accomplishing your goals. But they are simply references. They can almost certainly be improved upon or even ignored all together in favor of other approaches.

These batch files assume the GNUWin32 CoreUtils package is installed (for the sort, uniq and comm utilities). The example which uses comm also requires sed (a separate package).

For completeness, the following (minimal) text files were used as test input for the example scripts outlined below:

ex. dup1.txt

abc
def
123
ghi

Unique Line: 123

--

ex. dup2.txt

ghi
abc
jkl
def

Unique Line: jkl

Example Batch - Single File Output

In conjunction with e.g. the GNUWin32 CoreUtils package, you can create a Windows batch file that supports dragging/dropping two text files onto it, then having the non-duplicated lines displayed as a third, single file opened in ex. Notepad++:

ex. nodupes.bat

@ECHO Off

@REM Two input files (paths)
gnu-sort %1 %2 | uniq -u > uniq_text.txt

@REM Check our command output.
:: PAUSE

Note that in the example above, gnu-sort is the GNUWin32 sort utility, renamed to avoid conflicts in Windows (see the Windows Conflicts section).

While this batch file can be used in a "drag and drop" mode for convenience, you can also pass the relevant file names directly at the command line e.g. nodupes.bat dup1.txt dup2.txt. This is true for the other batch examples in this section as well.

Example Batch - Single File Output (Notepad++)

To achieve something similar more directly in Notepad++, you can create a new batch file similar to the one already described above but with a single input:

ex. nodupes-npp.bat

@ECHO Off

@REM One input file (path)
gnu-sort %1 | uniq -u > C:\path\to\uniq_text.txt

You would then save this .bat file in a static location for Notepad++ to reference. Then, in Notepad++ e.g. under the Run... menu ( F5 ), you could invoke this batch file for whatever (single) file was currently open:

cmd /c C:\path\to\nodupes-npp.bat "$(FULL_CURRENT_PATH)" A couple caveats exist to this approach, however: 1) The current file open would need to be a combined version of the two files you wished to compare data from. 2) The file you wish to apply this to must be saved before using this .bat/Run... approach. This is because: • "$(FULL_CURRENT_PATH)" isn't available in Notepad++ until the file is saved.

• If changes are made to the file, any updates need to be applied to the actual file being accessed (i.e. the version on disk, not the version currently open in Notepad++).

Example Batch - Multiple File Output

If you want unique lines divided (relatively) by their originating files, you can instead use a combination of comm and sed:

ex. nodupes-separated.bat

@ECHO Off

@REM Use sed to divide our comm -3 output into two files,
@REM using a simple sed script (filter.sed).

gnu-sort %1 > sorted_duplicates_1.txt
gnu-sort %2 > sorted_duplicates_2.txt

comm -3 sorted_duplicates_1.txt sorted_duplicates_2.txt | sed -f filter.sed

@REM Check our command output.
:: PAUSE

@REM Clean up our temporary files.
DEL sorted_duplicates_1.txt
DEL sorted_duplicates_2.txt

@REM These file names are defined in ex. filter.sed

ex. filter.sed

/^\t/ {
s///
w uniq_text_2.txt
d
}
/^[^\t]/ {
w uniq_text_1.txt
d
}

Note that while comm is part of the CoreUtils package, sed is a separate download with GNUWin32. Keep in mind that the batch file above won't alter your original files, it will create two new ones (so renaming them is up to your). Also, as written, ex. filter.sed need to be in the same directory as your batch file (e.g. nodupes-separated.bat).

Windows Conflicts (GNUWin32)

GNUWin32 contains some utilities with same name as Windows utilities or commands (e.g. sort). This can present a (mild) issue, as each utility may operate differently. The conflicts I am personally aware of in the GNUWin32 CoreUtils package are:

• date
• dir
• echo
• find
• hostname
• sort
• whoami

Keep in mind there may be other conflicts in separate packages as well, e.g. tree.

If you want to access the GNUWin32 versions from your environment path (i.e. without typing their full path), you can either rename the appropriate GNUWin32 executables themselves (recommended) or use surrogate batch files whose names are different (less desirable). A surrogate batch file would simply contain e.g.:

ex. gnu-utility.bat

@ECHO Off
C:\path\to\GNUWin32\bin\utility.exe %*

Using Windows Utility Versions

You may be able to use the Windows version of a utility, but they can require reworking how you approach things. For instance, Windows sort by default can only take one input at a time, so the original nodupes.bat example above might look like the following when using it:

ex. nodupes-win-sort.bat

@ECHO Off

@REM Helps compensate for a minor issue with cat and newlines
ECHO. > blank

@REM Using Windows sort, not GNUWin32 sort
cat %1 blank %2 | sort | uniq -u > uniq_text.txt

@REM Clean up our temporary file.
DEL blank