Tell me more ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.
up vote 0 down vote favorite

My Linux machine has 16 processors and free memory of 70 GB but when I run an egrep command, the command is using (checked through the top command) 100% of CPU and 0.5 of memory:

22391 aime      25   0 22.5g 337m  880 R 100.1  0.5   1094:06 egrep

That means that only 1 out of 16 CPUs are used.

How can I improve the performance here or make my egrep use more CPUs?

I am actually grepping in the directory structure which is 40 GB in terms of data size.

The approximate regex is :

egrep -H -i -R -a -o -n ([[:alnum:]_\.]+@[[:alnum:]_\.]*(xyz|abc)\.com) <dir>

Also , since I am running this command through Perl script, can ForkManager help me in this regard ?

Thanks

share|improve this question
2  
Split the file into separate segments and run separate egrep commands against each segment? How big of a file are you processing? Are you not I/O bound anyway? What is the regex you're using? – Jonathan Leffler Jan 17 '12 at 7:30
Please edit your post to include info Jonathan has asked for. good luck. – shellter Jan 17 '12 at 7:38
"The approximate regex"? Are you not sure? – Johnsyweb Jan 17 '12 at 8:33
That is because I have modified it here before putting . – Pargarg Jan 17 '12 at 8:38
This question is better suited to unix.stackexchange.com – Zaid Jan 17 '12 at 8:52
show 1 more comment

migrated from stackoverflow.com Jan 17 '12 at 12:57

2 Answers

up vote 2 down vote

The CPU utilization issue is rooted in threading. egrep = 1 thread => 1 cpu. If you want to use more CPUs, then you need to split up the task into 16 smaller tasks. I suggest dividing the data into 16 chunks and running one egrep on each, and make sure they're assigned to different CPUs each.

share|improve this answer
How to make sure that each egrep is assigned to different CPU ? I am firing the egrep command through a PERL script ? – Pargarg Jan 17 '12 at 8:20
You should leave that to the Linux kernel (or whatever OS you are using). If you create one process for each CPU, the load will in most cases be split evenly among the CPUs. – Automaton Jan 17 '12 at 8:40
What Krumelur said. However, this being such big operations, you might want to set the afinity manually. How to do this depend on the linux flavour of choice. – jarmund Jan 17 '12 at 8:55
Look into the -P flag of xargs as answered in stackoverflow.com/questions/5200591/how-to-use-grep-efficiently – Schwern Jan 17 '12 at 9:11
up vote 1 down vote

Ideas:

  • If you're using a Perl script anyway, why not use Perl to do the matching?
  • From a shell script, find plus GNU parallel (or xargs -P, as Schwern writes) are the most obvious way of applying egrep on your input files in parallel; they can obviously be invoked from Perl as well, or you can use Perl equivalents (e.g. File::Find).
  • You can parallellize by piping greps: the first to do very quick, simple pre-filtering, the next to do exact, expensive matching - but whether this approach can save you any time depends on what you need to match and on egrep implementation details that I'm not familiar with.
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.