Questions tagged [uniq]
The uniq tag has no usage guidance.
26
questions
23
votes
4
answers
34k
views
Is there a Windows equivalent to the Unix uniq?
I need remove duplicate lines from a text file, it is simple in Linux using
cat file.txt |sort | uniq
when file.txt contains
aaa
bbb
aaa
ccc
It will output
aaa
bbb
ccc
Is there a Windows ...
- 1
0
votes
2
answers
628
views
remove duplicated lines taking into account only two columns
I have a file that has some duplicated lines if I take into account only the two first columns.
For example
2 2 7 7
1 1 3 3
1 1 4 4
1 1 5 5
1 1 6 6
3 3 7 7
4 4 9 9
4 4 2 2
In this example, I have ...
- 5,581
6
votes
3
answers
7k
views
Remove non-duplicate lines in Linux
how can I remove non-duplicate lines from text file using any linux program linke sed, awk or any other?
Example:
abc
bbc
abc
bbc
ccc
bbc
Result:
abc
bbc
abc
bbc
bbc
Second list have removed ccc ...
- 973
2
votes
1
answer
69
views
Count based on unique subset of fields
I have a text file that is structured as follows:
P,ABC,DEF
P,GHI,JKL
B,ABC,DEF
B,MNO,PQR
I want to get a count of how many times a line appears where fields 2 and 3 are the same while preserving ...
- 74.3k
0
votes
0
answers
25
views
Find different file in directory
I try find one file that is different on folder.
I have 4 files i.e. file1, file2, file3 and file4. File from 1 to 3 have text "not uniq", file 4 is empty. How can i find this different file?...
- 123
0
votes
1
answer
572
views
Sort lines in file by first field treated as number in shell
I have a file with the format shown below (a number + an IPv4 address). This format was generated by using uniq -c. I want to sort the lines after the first number in each of them. I would've gone for ...
- 26.4k
4
votes
1
answer
6k
views
Grep on macOS: find unique occurrences of a capturing group in regular expression
I am on macOS and would like to get into using grep (or a similar tool) to find unique occurrences of a certain pattern in a codebase. For example, for finding all console.somemethod() calls in ...
- 17.9k
-1
votes
2
answers
605
views
Sorting files by “line content” frequency - print duplicates
Imagine there is a file -
a
b
b
b
b
c
c
d
d
d
I want the output to be sorted based on frequency (I want the duplicate lines to be printed as well) as -
b
b
b
b
d
d
d
c
c
a
- 20k
-1
votes
2
answers
166
views
find number of teams from a file [closed]
I have started learning bash very recently and have to find the number of teams from a file, the file can be downloaded from this link.
Below is a sample
I understand that I have to use cut and uniq ...
- 46
-1
votes
1
answer
649
views
How can I filter unique lines based on a substring?
If I have a set of URLs in a file like:
http://example.com/do?foo=bar&etc=123
http://example.com/do?etc=456&foo=bar
I want to filter this out so that there's only one url for each unique ...
- 4,190
1
vote
1
answer
3k
views
Printing unique lines with uniq, OS X Yosemite
I am trying to print the unique entries in a column of a .csv file. For this, I tried the following:
awk -F "," '{print $6}' dataCoarse.csv | uniq -u
which just prints the 6th column as it is. There ...
CommunityBot
- 1
3
votes
2
answers
489
views
How do I make uniq only consider the first field?
I am using FreeBSD 3.2-RELEASE
If I have some sorted text, like this last output—
zikla13:Oct:20:22:34
zikla13:Oct:5:00:31
zikla14:Oct:17:22:01
zikla14:Oct:12:23:35
zikla14:Oct:12:23:34
zikla14:Oct:...
- 3,883
0
votes
1
answer
2k
views
Remove list of items from another file in bash
What would be the most efficient method (no repeated command execution) to remove items listed in one file from another file (unordered) ?
One can easily get the list of non-matching items in the ...
- 26.4k
1
vote
1
answer
1k
views
Grep matching lines but return matching key IDs in GnuPG output
Having this input:
rsa2048/C7927B82 2015-08-30
rsa2048/FB2D99F9 2015-08-30
I want grep to only return the key ID, for example: C7927B92. Using the pattern from Grep characters before and after match?...
CommunityBot
- 1
0
votes
1
answer
868
views
option -f for linux command
this is the content of test file:
this this
this that
is that
a that
MAP that
map that
when I do uniq testCase -f1, the result is:
this this
this that
which is reasonable, but if ...
- 65.2k
0
votes
2
answers
163
views
Sort on ID, then sort on date, then remove lines which have the older date?
I have a couple of tab delimited files.
Each file is structured like so:
ID Title Rating Date_Rated
What I want to do is merge all these files into one, ...
- 458
0
votes
3
answers
379
views
Filtering 2nd field from a data set and then using uniq on the output
I've got a dataset that goes like this:
AAAAA 11111 Data1
AAAAA 11111 Data2
AAAAA 11111 Data3
AAAAA 11112 Data4
AAAAA 11112 Data5
AAAAA 11112 Data6
AAAAA 11112 Data7
AAAAA 11113 Data8
AAAAA 11114 ...
- 6,244
0
votes
1
answer
113
views
gnu `uniq -c`: display counts graphically as bars
With sort $file | uniq -c, it's possible to sum up files/input and show the number occurences like that:
13 foo
1 bar
To get a better grasp of this data, I'd like to see them graphically ...
- 8,039
0
votes
1
answer
166
views
Sort piped to uniq unexpected output
I have a file that looks like this:
"string",192.168.1.0,,TX,192.168.1.2,,,100,0,0,10,10,10,,
"string",192.168.1.0,,TX,192.168.1.1,,,100,0,0,10,10,10,,
"string",192.168.1.0,,TX,192.168.1.4,,,100,0,0,...
- 2,646
1
vote
3
answers
4k
views
remove lines with duplicate words
I have a sorted file with lines like this
word1 abca
word1 abcb
word1 abcc
word2 abca
word2 abcb
word3 abbb
...........
and i want to have something like this
word1 abca
word2 abca
word3 ...
- 53.4k
0
votes
1
answer
724
views
How to print statistics from lines in a file
I often find myself computing percentages manually after filtering through a log file to find the proportion of X or Y in it. Can this easily be achieved via common CLI tools?
- 135
2
votes
1
answer
851
views
Counting duplicates lines from a stream
I'm currently parsing apache logs with that command:
tail -f /opt/apache/logs/access/gvh-access_log.1365638400 |
grep specific.stuff. | awk '{print $12}' | cut -d/ -f3 > ~/logs
The output is a ...
- 131
7
votes
3
answers
3k
views
Remove duplicates in each line of a file
How can I remove duplicates in each line, for example here?
1 1 1 2 1 2 3
5 5 4 1 2 3 3
I'd like to get this output:
1 2 3
5 4 1 2 3
There are lots of lines (100,000) and in each line I want ...
- 26.4k
0
votes
2
answers
6k
views
linux sort -n uniq -c
I have a text file which contains about 50MB of text:
valA
valB
valC
valZ
valB
Whats the quickest way to:
cat the file
Display the unique appearences
- 419
1
vote
1
answer
542
views
gnu sort/uniq: sorting by number of times
How can I use GNU sort and uniq to have the most common occurrences on top instead of numerical or alphanumerical sorting? Example list.txt:
1
2
2
2
3
3
Since '2' occurs 3 times, should be on top, ...
- 27.8k
0
votes
2
answers
204
views
lines identical except for their first word, put the first word in a variable
I have file(sudoers) in which lines can be identical except for the first word in them (different users can execute the same set of commands). I am able to extract such a line using the command:
# ...
- 18.8k