1

I have a network cluster of about 100 machines, a piece of software and a large list of different parameters for the software.

I want to use the cluster to speed up the calculation, so each machine is supposed to run the software with a different set of parameters from the list. When a machine finishes its calculations it should receive the next set of parameters and run the software again until all of the parameters have been used.

Is there a way to do this in a bash script via ssh? I suppose pssh is the way to go, but I couldn't figure out how to do it.

  • 1
    openmpi or openmp is used for developing distributed algorithms on clusters. OpenMPI has bindings for different languages, but I don't think there will be some for bash, since you usually need high performance in parallel environment. Basic communication is made using SSH there. I certainly don't recommend doing it on your own without any library. – Jakuje Jul 23 '15 at 11:11
  • I guess openmpi would be overkill in my scenario. In fact, the algorithm is not distributed at all. It just has to be run with all these different sets of parameters, but the individual runs are independent of one another. So I would imagine it as follows: Run the software on all machines, keep track which machines return and then instantly feed them new parameters. – Briensturm Jul 23 '15 at 11:22
  • I'm with @Jakuje on this. You're trying to implement how workload should be distributed among machines. Unless you have academic reasons or whatnot, it is not a good idea to do this on your own since a) it has already been done in different frameworks (Ømq, GridEngine...) and b) you'll probably come short by doing this on your own, thus ending using an already existing tool. – Felipe Lema Jul 23 '15 at 12:29
  • my simple algorithm: first, setting up a samba/nfs share so all nodes can r/w from it, then write required parameters to separate files on that shared directory. Each node pulls a file to get necessary parameters, run the program and (re)move the corresponding file. When program finishes pull a new file until no more file at that shared directory. – Chris.C Jul 23 '15 at 18:21
2

I have written a bash script which does a kind of similar thing, exhausting local CPU cores. When a core frees, it calls a new calculation, until the calculations are finished. I also have a bit of experience scripting bash with ssh (it requires a passwordless ssh-key, if you are comfortable with that security risk). This is a personal example out of context, but the idea is that it is a bash script which loops dynamically based on processing time and varies parameters. The $CORES variable, in your case, needs to be filled with available servers, and we need to find a way to track them to know which to call next.

    Loop () {
      # looping function over all runs with the same header, multi-threaded per core.
      CMDINIT="$CMD"
      for i in "$TREEIN"/"$NAME"*.root  # loop over all the existing raw runs of that name
      do
        # name of the output file and path, eg Tree/30s_production/30s_production-1001.root
        OUTPUT=`echo $i | sed "s#$TREEIN/##" |sed "s#$NAME#$TREEOUT/$NAME/$NAME-#"  `
        INFILE=`echo $i | sed 's$.*/$$'` # name of the input file name, eg 30s_production1001.root
        if [ ! -e  $OUTPUT ];then # only run if the output file does not exist (won't overwrite existing data)
          if [ ! $Cflag ];then # only call the program if we aren't cleaning 
            echo "Outputting to $OUTPUT..."
            RUNNO=`echo $INFILE | sed "s/$NAME//" | sed 's/\..*//'` # get the run number
            # there is a way to do this using the Run function?  Seems trickier with backgrounding, getting PID, and so on
            CMD="$CMDINIT -R $RUNNO"
            printf "Executing run with the command:\n\t$CMD\n"
            $CMD & PIDS="$PIDS $!" # call run on the run number in background w/o renice
            #$CMD & PIDS="$PIDS $!" && sudo renice -n 0 -p $! # call run on the run number in background, renice to -10
            while [ `jobs | wc -l` -eq $CORES ] # only run one run command per core
            do
                    jobs > /dev/null # without this the while loop doesn't seem to refresh?
                    sleep 1 # keep waiting until run is not running on a core
            done
          fi
        else # the output file exists -- should never happen as we check NeedClean first, but anyway, safer
          echo "$OUTPUT exists, please run clean!"
          exit 1
        fi
      done
    }

There are two 'clever' parts here (or 'hacks' if you prefer). One is the while loop that checks the number of jobs and waits until something is free. (My present do loop is based on one parameter, but this is easy to adjust.) Those jobs are the ones inside the bash script, and this is how it attains the concept to loop again on a condition that a job is completed; keep in mind this will be identical whether the job is local or remote: A command to call an SSH command will be the same as a local job (though, we may need to collect all the results later from all the servers, or have the servers writing back the data locally, etc, as needed). Another critical aspect for me is how when $CMD is called, it also tacks the process number onto a counter called $PIDS. This allows a control_c trap on the local bash script to be able to kill all child processes, which would include spawned processes on all of your 100 servers, should you decide to terminate early; the consequence of not tracking this is about as horrendous as you can imagine!

If you want to check the main script, it is here: https://github.com/goatface/crabat/blob/master/crabat

We need to modify the definition of the variable $CMD to be something on the order of

CMD=ssh '$USER@$HOST' /path/to/executable

After this we should dynamically tack on the flags to the executable to control the different parameters (we could also push these in a text file to each server via scp, but finally we need to track them anyway, and the difference is not significant for me). My case sets most of the parameters once, but there is no reason we cannot call this each time a server open. It looks like this, where I use flags, but this is trivial to set to spool sets of text files with parameters. Increment a counting variable on an awk field for each field, in order, until is exhausted, etc, and reset it in the Loop function each time the next counting variable is incremented, sequentially through all permutations of parameters.

    SetFlags () {
      # base command
      CMD="./run"
      # add options based on final flags
      [ $Rflag ] && CMD="$CMD -R $Rval"
      [ $Bflag ] && CMD="$CMD -B"
      [ $Hflag ] && CMD="$CMD -H $Hval"
      [ $Iflag ] && CMD="$CMD -I $Ival"
      [ $rflag ] && CMD="$CMD -r"
      [ $dflag ] && CMD="$CMD -d"
      [ $sflag ] && CMD="$CMD -s"
      [ $xflag ] && CMD="$CMD -x"
      [ $pflag ] && CMD="$CMD -p"
      [ $tflag ] && CMD="$CMD -t"
    }

I have enabled to receive emails upon replies, since I find this to be an interesting question, but I need more time to think about tracking the hosts as they deallocate. Sorry I have not answered the whole question, yet!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.