20

I need remove duplicate lines from a text file, it is simple in Linux using

cat file.txt |sort | uniq

when file.txt contains

aaa
bbb
aaa
ccc

It will output

aaa
bbb
ccc

Is there a Windows equivalent? or how do this in a Windows way?

3
  • 10
    On Unix, you could write it as sort -u file.txt
    – jfs
    Apr 23, 2018 at 6:24
  • 1
    There is also WSL which works pretty well as far as this sort of stuff goes Apr 23, 2018 at 13:07
  • Maybe you want to set something as solution, if you don't have any further questions? May 9, 2018 at 14:28

4 Answers 4

38

The Sort-Object cmdlet in PowerShell supports a -Unique switch that does the same thing as uniq:

Get-Content file.txt | Sort-Object -unique

Of course, owing to presence of aliases in PowerShell, you can also write:

type file.txt | sort -unique

Additionally, there is an undocumented /unique switch in sort.exe of Windows 10, so, this should work in Command Prompt:

type file.txt | sort /unique
11
  • 2
    I don't think the Windows command (sort.exe) supports this; it looks like a feature of the PowerShell builtin.
    – Ben Voigt
    Apr 23, 2018 at 4:11
  • 1
    type unsorted.txt | sort -unique > sorted.txt This really work under win10 and writed unique values to new file
    – Lixas
    Apr 23, 2018 at 5:52
  • 7
    @BenVoigt surprisingly, type file.txt | sort /unique works with undocumented switch /unique of sort.exe utility (at least on Windows 10). On the other side, you are right that provided example is PowerShell Get-Content file.txt | Sort-Object -unique, in fact.
    – JosefZ
    Apr 23, 2018 at 5:57
  • 2
    sort /unique errors with Invalid switch. on Windows 7 Enterprise. Apr 23, 2018 at 12:00
  • 4
    @JosefZ Well, I opened a bug for that, and now it is documented 😇 Jan 9 at 14:48
6

There's ports of uniq that work identically to the gnu/coreutils versions. I personally use the variation from GOW but git for windows has a significantly newer version. No cygwin required though for the latter you need to look in /usr/bin

Since these packages also contain cat, sort and uniq - your workflow should be mostly identical, and cat file.txt |sort | uniq should work mostly identically

2

You can easily write the command "uniq" by yourself. Save this in a batch file "uniq.cmd" somewhere in your %path% can find it (e.g. in %windir%\system32). This version is NOT case sensitive:

@echo off
setlocal DisableDelayedExpansion
set "prev="
for /f "delims=" %%F in ('sort %*') do (
    rem "set" needs to be done without delayed expansion
    set "line=%%F"
    setlocal EnableDelayedExpansion
        set "line=!line:<=<!"
        if /i "!prev!" neq "!line!" echo(!line!
        set "prev=!line!"
    endlocal
)

This works with "uniq mytextfile" as well as "cat mytextfile | uniq"; as all input and arguments are simply passed to the sort command.

Starting with Windows 7, you may want a really case sensitive version (the difference ist undocumented switch "sort /C" and no "if /i"):

@echo off
setlocal DisableDelayedExpansion
set "prev="
for /f "delims=" %%F in ('sort /C %*') do (
    rem "set" needs to be done without delayed expansion
    set "line=%%F"
    setlocal EnableDelayedExpansion
        set "line=!line:<=<!"
        if "!prev!" neq "!line!" echo(!line!
        set "prev=!line!"
    endlocal
)
2
1

Addition to Yu Jiaao's answer. You can invoke the sort-object powershell cmdlet in a command prompt like:

type file.txt | powershell -nop "$input | sort -unique"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.