What's the simplest way to remove lines from one file matched with lines from another file? For example, if I have the following files:
file1.csv:
[email protected]
file2.csv:
1,[email protected],somehash1
2,[email protected],somehash2
3,[email protected],somehash3
As a result I'd like to have file3.csv:
1,[email protected],somehash1
3,[email protected],somehash3
What's the fastest way to solve this task? These files are a few GB in size.
grep -v -F -f file1.csv file2.csv > file3.csv seems the simplest. But you should do performance tests with smaller files first. (I agree with soandos' comment that such big files might need a dedicated solution.)
cat file2.csv | fgrep -vf file1.csv > file3.csv
cat. You could also use < file2.csv | fgrep -vf file1.csv > file3.csv.
awk -F, '
FILENAME == ARGV[1] {to_remove[$1]=1; next}
! ($2 in to_remove) {print}
' file1.csv file2.csv > file3.csv
You have to have enough memory to read in file1 at once.
Here's another option: join
$ join -t , -v 2 -1 1 -2 2 file1.csv file2.csv
[email protected],1,somehash1
[email protected],3,somehash3
However, from the man page "Important: FILE1 and FILE2 must be sorted on the join fields." so factor that into your decision.
You could loop over each line in file1, and grep matching lines out of file2?
cp file2.csv file3.csv
cat file1.csv | while read line; do
grep -v ${line%?} file3.csv > temp.csv
cat temp.csv > file3.csv
done
rm -f temp.csv
Untested.
Edit: Tested, seems to work OK. Just make sure you have a trailing newline in file1.
set -x and set -v and saw that a newline was on the end of the string being grep'd for, so I added the %? and it worked... And you are correct that this processes file2 once for each line in file1, but surely the only other approach would be to read file1 once for each line in file2? The OP said the files were 'few gigs each', so I'm not sure how much difference there is between file sizes.
Jul 26, 2011 at 1:05
Does file1.csv have to stay unmodifieed?
sed 's|.*|/^&.*/d|' file1.csv > file1.sed
sed -f file1.sed file2.csv > file3.csv
I don't know how much memory it consumes. AFAIK, it will test the whole -sed file each time on the whole input (2.csv).
If the input is sorted, and the patterns are sorted too, you could implement a faster solution.
Make sure file3.csv exists (and it's empty)
echo > file3.csv
diff file1.csv file2.csv | patch file3.csv
Et voilá!
This doesn't need a 'coded solution.' If you sort the lines first, the algorithmic complexity is reduced by several orders of magnitude.
See this answer for better performance, both in terms of CPU time and memory:
https://stackoverflow.com/questions/4366533/remove-lines-from-file-which-appear-in-another-file
