I have 2 large text files (~500M, ~15GB ea) That look like this:

FileP.txt:

test@test.com:testtest 
test@test.com1:testtest1
test@test.com2:testtest2
test@test.com3:testtest3
test@test.com4:testtest4

FileE.txt

test@test.com:testtest
test@test.com0:testtest0
test@test.com2:testtest2
test@test.com3:testtest3
test@test.com5:testtest5

(Notice how FileE.txt has lines that aren't in FileP.txt. I don't want those included. Bold lines are lines that should end up in output.txt, as they're not in FileE.txt.)

I want to run FileE.txt against FileP.txt and remove all the lines that were found in FileE.txt from FileP.txt and output into a new file.

It should look like this:

Output.txt:

test@test.com1:testtest1
test@test.com4:testtest4

I tried a few commands,

Here's my grep command:

$ grep -Fvxf FileE.txt FileP.txt > output.txt

However, I get this error (Obviously because the files are too large):

grep: memory exhausted

For those interested, running $ ulimit -a returns:

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 8
stack size              (kbytes, -s) 2032
cpu time               (seconds, -t) unlimited
max user processes              (-u) 256
virtual memory          (kbytes, -v) unlimited

So my question is, what would be the most efficient & easiest way to complete this process?

NOTE: Files are not sorted.

  • Are the files guaranteed to be sorted? Please do not respond in comments; edit your question to make it clearer and more complete. – Scott Jul 17 at 23:44
  • 1
    Sort the files, do a diff FileP.txt File.txt and keep only the lines prefixed with <? – xenoid Jul 18 at 0:08
  • @Scott Done. How would I keep lines only prefixed with <? I've never used Diff. Does it make one large folder with < prefixed to the lines with the difference? Does this also keep the lines that are available in FileE.txt that don't exist in FileP.txt? – AndrewWilliams Jul 18 at 0:21
  • Another idea? Get them into a MySQL database. With data this large, a plain text file is never a great option. Even if that MySQL database is only considered a huge cache. That said, if you need to stay in the command line world I feel your pain. – JakeGould Jul 18 at 2:23
  • @JakeGouldI thought about that, honestly. I have to stay in CL :( – AndrewWilliams Jul 18 at 5:17
up vote 0 down vote accepted

If the files are sorted, do

comm -23 fileP.txt fileE.txt

comm compares two sorted files, looking for lines that they have in common.  For example, given this file of colors whose names begin with consonants:

blue
green
purple
red
white
yellow

and this list of colors whose names end with vowels:

blue
indigo
orange
purple
white

the command comm colors1 colors2 produces this output:

                blue
green
        indigo
        orange
                purple
red
                white
yellow

where:
the first column contains colors that begin and end with consonants (in colors1 but not colors2), the second column contains colors that begin and end with vowels (in colors2 but not colors1), and the third column contains colors that begin with consonants and end with vowels (in both colors1 and colors2).  For your files (shown in your question), comm fileP.txt fileE.txt produces

                test@test.com:testtest
        test@test.com0:testtest0
test@test.com1:testtest1
                test@test.com2:testtest2
                test@test.com3:testtest3
test@test.com4:testtest4
        test@test.com5:testtest5

The options are a little non-intuitive: -23 means suppress the second and third columns, showing only the first (lines that are in the first file but not the second).  So,

$ comm -23 fileP.txt fileE.txt
test@test.com1:testtest1
test@test.com4:testtest4

Note that this will not work correctly if the files are not sorted.  It the files aren’t sorted, sort them.

To enlarge on a comment by xenoid, sort the files and type

diff fileP.txt fileE.txt | sed -n 's/^< //p'

The output of diff shows the lines that are in only the first file, preceded by , and the lines that are in only the second file, preceded by .  The sed selects only the lines that begin with and strips it off.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.