Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

If I want to process large number of files with command "do_something" which only can use one core, what's the best way to use all available cores assuming each file can be processed independently?

At this moment I do something like this:

#!/bin/zsh
TASK_LIMIT=8
TASKS=0
for i in *(.)
{
  do_something "$i"&
  TASKS=$(($TASKS+1))
  if [[ $TASKS -ge $TASK_LIMIT ]]; then
    wait; TASKS=0; fi
}
wait

Obviously, this is not efficient because after reaching $TASK_LIMIT it waits when all "do_something" finish. For example in my real script I make use of about 500% of my 8-core CPU instead of >700%.

Running without $TASK_LIMIT is not an option because "do_something" may consume lots of memory.

Ideally, the script should try to keep number of parallel tasks at $TASK_LIMIT: for example if task 1 of 8 finished and there is at least one more file to process, the script should run next "do_something" instead of waiting for remaining 7 tasks to finish. Is there a way to achieve this in zsh or bash?

share|improve this question
    
hint: use trap to catch SIGCHLD in monitor mode. –  Keith Oct 26 '12 at 16:22

2 Answers 2

up vote 5 down vote accepted

I strongly suggest having a look at GNU parallel. It does exactly what you want and doesn't depend on any particular shell.

share|improve this answer

Remember how many processes you started. When a process ends, decrease the count. When the count is lower than the maximum, start a new process.

The only problem is how to signal the end of a process. You can e.g. create an emty file of a given name in /tmp (composed of $$ and $BASHPID).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.