0

I am trying to clean a Huge database of emails, but when I use the following Grep command, I get "Memory Exhausted" Error. 

pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' EMAILS.txt) > RESULT.txt

The data set in both file are really huge like more than 5 Million emails, therefore the Linux server gives me "Memory Exhausted" Error.

Therefore I decided to split the file in into small files with 10,000 emails each to process.

split -d -l 10000 EMAILS.txt Segment

How to create a Bash Script with a loop, where it checks for the created split file in increasing order and starts processing from the first file created - Eg: Segment00

pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' Segment00) > RESULT.txt

...then automatically loop the same command and substitute the second segment file in the command - Eg: Segment01

pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' Segment01) > RESULT.txt

...carry the loop until the last segment and then finally terminate.

Can you please help me write such Bash script? I cannot figure out how to automatically substitute the segmented files in increasing order automatically in the loop to write a shell script.

Please help.

Improve this question

1 Answer 1

Reset to default
1

You can use for example for loop like this:

for i in Segment??
do
pv "Suppresion-List.txt" | grep -vf <(sed 's/^/^/; s/$/$/' "$i") >> RESULT.txt
done

This will exec the command with all Segment<symbol><symbol> files and ADD result to RESULT.txt

Based on the comment the command should be:

pv "RESULT-1.txt" | grep -vf <(sed 's/^/^/; s/$/$/' "$i") >> RESULT.txt
Improve this answer
12
  • Thank you. So if the segment file contains 3 numbers (Eg: Segment001, then I should use [???] thrice?
    – Joney Walker
    Feb 8, 2019 at 7:11
  • 1
    Right. I use such way to restrict the filenames. You can use also Segment* if you are sure only files in question start with this string
    – Romeo Ninov
    Feb 8, 2019 at 7:13
  • I tested this, there is an issue when I ran this... The final Result file same as the Suppression-List file. What I figured was, in the SECOND ROUND, actually, the Suppresion-List.txt should be replaced by the RESULT-1.txt of the first round output. Only then, we will have the desired result at the end. So how do I do that, Sir? From the Second Round, we need to automatically add the previous round Result file in the place of Suppresion-List.txt and goes on...
    – Joney Walker
    Feb 8, 2019 at 16:56
  • @JoneyWalker, just replace the string after pv command. If you do not mean this please clarify editing your question
    – Romeo Ninov
    Feb 8, 2019 at 16:58
  • 1
    Ok Sir, I will create a New Question and Tag you in that...
    – Joney Walker
    Feb 8, 2019 at 17:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .