Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm maintaining a data-warehousing system that involves a lot of dependent jobs (data import, transform, etc). I've been using Linux's crontab to manage them until the dependency between jobs get complicated.

Basically I'm looking for some cron replacement that helps me with the following scenario:

  • Run job A at 00:05 (easy). Usually this is the import job.

  • Schedule job B, C, D to run after job A finishes. Job D only runs 30 mins after job A finishes (to distribute load). These are the transform jobs.

  • Job E run when all B, C, D finish. Usually this is the job that bring aggregated data to a web front-end database.

All of these happen on the same node.

I imagine it looks like a topology graph.

A--> B -------------->---> E
 \-> C -------------/   /
 \-> (delay 30mins) -> D

Are there such simple linux-based tools that support this? I've looked into Airbnb's Chronos but it seems overkill for my need.

Edit: The above scenario is just a simplified version of what's happening. We have a lot more daily jobs and the dependency is a lot more complicated. So I'm actually looking for some "cron on steroid" than case-by-case bash scripts to cater each scenario.

share|improve this question
    
I guess using the Jenkins CI server would be overkill as well. Still, it'd work. –  Daniel Beck May 13 '13 at 10:27
1  
You could do this with a fairly simple bash script that runs at 00:05 (through any crond), starts job A and waits for its completion, then starts B and C in the background as well as a backgrounded subprocess which waits 30 min then starts D, then waits for all of them to finish before starting E, then when E completes simply exits. Throw in a little error handling for good measure and you should be good to go. Possibly useful links: stackoverflow.com/q/356100/486504 stackoverflow.com/a/6041820/486504 Google "bash wait for subprocess" or similar. –  Michael Kjörling May 13 '13 at 11:36
    
Michael: Thanks. Though my use case is much more complicated than that. I've updated the question to reflect. –  huy May 14 '13 at 2:44
    
So you're looking for a simple tool to handle an unspecified more complex version of what you use as an example... Frankly, if so, I'm not sure the question as it stands is answerable. Then at least explain why the alternative you have found is "overkill" for your needs, and what specific features of it you do need. –  Michael Kjörling May 15 '13 at 13:20
add comment

2 Answers

As Michael Kjörling suggested in the comments, you should be able to do this with a simple bash script. Something like this:

#!/usr/bin/env bash

## Log file to which the "echo" commands bellow will write
logfile="/tmp/$$.log"

## Change "ls /etc >/dev/null " to reflect the actual 
## jobs you want to run but keep the  "&& echo job N finished" as is.
jobA="ls /etc >/dev/null"
jobB="ls /etc >/dev/null && echo 'job B finished' >> $logfile"
jobC="ls /etc >/dev/null && echo 'job C finished' >> $logfile"
jobD="ls /etc >/dev/null && echo 'job D finished' >> $logfile"
jobE="ls /etc >/dev/null";

## Run job A, launch jobs B and C as soon as A is finished
## and launch job D 30 minutes after A finishes.
eval $jobA && (sleep 30 && eval $jobD) & eval $jobB & eval $jobC &


## Now, monitor the logfile and run job E when the rest have finished
while true; do
    lines=`wc -l $logfile | cut -f 1 -d ' '`;
    echo "$logfile : $lines"
    ## The logfile will contain 4 lines if all jobs have finished
    if [ "$lines" -eq 3 ];
    then
      ## Run job E
      eval $jobE 
      ## Delete the logfile
      rm $logfile
      ## exit the script
      exit 0;
    fi
    ## Only check if the jobs are finished once a minute
    sleep 60;
done

If you use cron to launch this script at 00:05 it should do what you want. The main trick here is the use of subshels () and &&. Subshels allow you to run multiple background jobs and && to only run jobs once another job has exited successfully.

share|improve this answer
    
Thanks for your help @terdon. Sorry I wasn't more clear about my situation. The number of jobs I have are much more than that, and the inter dependency is a lot more complicated. The scenario I described is just a simplified version of what usually will happen. –  huy May 14 '13 at 2:39
add comment

BMC Software makes a product called Control-M which would be a perfect fit for your description of the issue. However, it isn't free :(

We use it to administer around 500 jobs in production, and somewhere close to 400 in test environments. You install clients on whatever machines you need, then set up jobs on the Control-M server to run on the clients. There are a whole lot of configurable parameters and scheduling criteria, all of which can be administered through a GUI or command line. The most fitting part to your problem is that it thrives on setting up input/output conditions for jobs so that you can have dependencies just by drag and dropping between jobs. We use it to set up workflow streams of 20+ jobs at a time.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.