0

I am trying to run 20 files at a time, using warc.sh or a better way to put it is 20 instances of warc.sh at a time with directory contents as the input. I am also trying to watch the PIDs in case bad things might happen, and load a new instance with a new input each time an old instance terminates.

# test if files exists and run
if [ "$(ls -A ${base_dir}/schedule)" ]; then

    # function to be backgrounded
    track()
    {
      ./warc "${base_dir}/schedule/$1 printf" "\nFinished: %d\n" "$1"
    }

    start=$(date '+%s')

    # Make an associative array in which you'll record pids.
    declare -A pids

    # Background an instance of the track() function for each number, record the pid.
    for n in `ls -A ${base_dir}/schedule`; do
        if [ ${#pids[@]} > 20 ]; then
          track $n &
          pid=$!
          echo "Backgrounded: $n (pid=$pid)"
          pids[$pid]=$n
        else
            echo 'Sleeping' ; sleep 2
        fi
    done

    # Watch your stable of backgrounded processes.
    # If a pid goes away, remove it from the array.
    while [ -n "${pids[*]}" ]; do
      sleep 1
      for pid in "${!pids[@]}"; do
        if ! ps "$pid" >/dev/null; then
          unset pids[$pid]
          echo "unset: $pid"
        fi
      done
      if [ -z "${!pids[*]}" ]; then
        break
      fi
      printf "\rStill waiting for: %s ... " "${pids[*]}"
    done

    printf "\r%-25s \n" "Done."
fi

Right now this is completely snowballing out of control and creating plus 900 background process =/

0

Write Makefile for it and run make -j 20.
You can write Makefile with shell script

  • I think you're misunderstanding what is going on – ehime Oct 28 '13 at 15:51
  • You need parallel processing and use some bash script for it, but your script doesn't work. I suggest don't reinvent already existing and well-working tools. – Alexander Kudrevatykh Oct 29 '13 at 8:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.