Removing Duplicate from File

Question

I have two files fileA and fileb

FILEA

1.1.1.1
2.2.2.2
3.3.3.3
4.4.4.4
5.5.5.5

FILEB

4.4.4.4
2.2.2.2

Output should be

1.1.1.1
3.3.3.3
5.5.5.5

Size of File A is around 60GB and Size of File B is 1GB

I used this command grep -vf FileB FileA > new.txt

But after sometime it gets stopped automatically and shows as Killed

Can someone please help me in this case.

Thanks in Advance

Answer 1

There are a few approaches you could take. If you're dealing with files of that size, just concatenating both files and only printing lines that appear once may be effective:

cat FILEA FILEB | sort | uniq -c | awk '$1==1{print $2}'
1.1.1.1
3.3.3.3
5.5.5.5

If the values won't be single "words" (i.e., you potentially have spaces and more fields on each line), then you could tell awk to print everything on the line after the uniq count of "1" on relevant lines:

cat FILEA FILEB | sort | uniq -c | awk '$1==1{print substr($0,index($0,"1 ")+2)}'
1.1.1.1
3.3.3.3
5.5.5.5

Of course, if you have lines in FILEA that may be duplicates within the file but you want to keep them because they don't appear in FILEB, that may require a different approach, like defining an array of values in FILEB and then checking each line in FILEA to see if it's in the array, like this:

awk 'BEGIN{ref="FILEB";while (getline < ref){remove[$0]};close(ref)}{if (!($0 in remove)){print}}' FILEA
1.1.1.1
3.3.3.3
5.5.5.5