23

I need remove duplicate lines from a text file, it is simple in Linux using

cat file.txt |sort | uniq

when file.txt contains

aaa
bbb
aaa
ccc

It will output 

aaa
bbb
ccc

Is there a Windows equivalent? or how do this in a Windows way?

Improve this question
3
  • 10
    On Unix, you could write it as sort -u file.txt
    – jfs
    Apr 23, 2018 at 6:24
  • 1
    There is also WSL which works pretty well as far as this sort of stuff goes
    – user2813274
    Apr 23, 2018 at 13:07
  • Maybe you want to set something as solution, if you don't have any further questions?
    – davidbaumann
    May 9, 2018 at 14:28

4 Answers 4

Reset to default
41

The Sort-Object cmdlet in PowerShell supports a -Unique switch that does the same thing as uniq:

Get-Content file.txt | Sort-Object -unique

Of course, owing to presence of aliases in PowerShell, you can also write:

type file.txt | sort -unique

Additionally, there is an undocumented /unique switch in sort.exe of Windows 10, so, this should work in Command Prompt:

type file.txt | sort /unique
Improve this answer
11
  • 2
    I don't think the Windows command (sort.exe) supports this; it looks like a feature of the PowerShell builtin.
    – Ben Voigt
    Apr 23, 2018 at 4:11
  • 2
    type unsorted.txt | sort -unique > sorted.txt This really work under win10 and writed unique values to new file
    – Lixas
    Apr 23, 2018 at 5:52
  • 7
    @BenVoigt surprisingly, type file.txt | sort /unique works with undocumented switch /unique of sort.exe utility (at least on Windows 10). On the other side, you are right that provided example is PowerShell Get-Content file.txt | Sort-Object -unique, in fact.
    – JosefZ
    Apr 23, 2018 at 5:57
  • 3
    sort /unique errors with Invalid switch. on Windows 7 Enterprise.
    – Don Cruickshank
    Apr 23, 2018 at 12:00
  • 4
    @JosefZ Well, I opened a bug for that, and now it is documented 😇
    – Tsahi Asher
    Jan 9, 2022 at 14:48
6

There's ports of uniq that work identically to the gnu/coreutils versions. I personally use the variation from GOW but git for windows has a significantly newer version. No cygwin required though for the latter you need to look in /usr/bin

Since these packages also contain cat, sort and uniq - your workflow should be mostly identical, and cat file.txt |sort | uniq should work mostly identically

Improve this answer
2

You can easily write the command "uniq" by yourself. Save this in a batch file "uniq.cmd" somewhere in your %path% can find it (e.g. in %windir%\system32). This version is NOT case sensitive:

@echo off
setlocal DisableDelayedExpansion
set "prev="
for /f "delims=" %%F in ('sort %*') do (
    rem "set" needs to be done without delayed expansion
    set "line=%%F"
    setlocal EnableDelayedExpansion
        set "line=!line:<=<!"
        if /i "!prev!" neq "!line!" echo(!line!
        set "prev=!line!"
    endlocal
)

This works with "uniq mytextfile" as well as "cat mytextfile | uniq"; as all input and arguments are simply passed to the sort command.

Starting with Windows 7, you may want a really case sensitive version (the difference ist undocumented switch "sort /C" and no "if /i"):

@echo off
setlocal DisableDelayedExpansion
set "prev="
for /f "delims=" %%F in ('sort /C %*') do (
    rem "set" needs to be done without delayed expansion
    set "line=%%F"
    setlocal EnableDelayedExpansion
        set "line=!line:<=<!"
        if "!prev!" neq "!line!" echo(!line!
        set "prev=!line!"
    endlocal
)
Improve this answer
2
1

Addition to Yu Jiaao's answer. You can invoke the sort-object powershell cmdlet in a command prompt like:

type file.txt | powershell -nop "$input | sort -unique"
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .