I have 2 large text files (~500M, ~15GB ea) That look like this:
email@example.com:testtest firstname.lastname@example.org:testtest1 email@example.com:testtest2 firstname.lastname@example.org:testtest3 email@example.com:testtest4
firstname.lastname@example.org:testtest email@example.com:testtest0 firstname.lastname@example.org:testtest2 email@example.com:testtest3 firstname.lastname@example.org:testtest5
FileE.txt has lines that aren't in
FileP.txt. I don't want those included. Bold lines are lines that should end up in
output.txt, as they're not in
I want to run
FileP.txt and remove all the lines that were found in
FileP.txt and output into a new file.
It should look like this:
I tried a few commands,
Here's my grep command:
$ grep -Fvxf FileE.txt FileP.txt > output.txt
However, I get this error (Obviously because the files are too large):
grep: memory exhausted
For those interested, running
$ ulimit -a returns:
core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited open files (-n) 256 pipe size (512 bytes, -p) 8 stack size (kbytes, -s) 2032 cpu time (seconds, -t) unlimited max user processes (-u) 256 virtual memory (kbytes, -v) unlimited
So my question is, what would be the most efficient & easiest way to complete this process?
NOTE: Files are not sorted.