Sign up ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I'm maintaining a data-warehousing system that involves a lot of dependent jobs (data import, transform, etc). I've been using Linux's crontab to manage them until the dependency between jobs get complicated.

Basically I'm looking for some cron replacement that helps me with the following scenario:

  • Run job A at 00:05 (easy). Usually this is the import job.

  • Schedule job B, C, D to run after job A finishes. Job D only runs 30 mins after job A finishes (to distribute load). These are the transform jobs.

  • Job E run when all B, C, D finish. Usually this is the job that bring aggregated data to a web front-end database.

All of these happen on the same node.

I imagine it looks like a topology graph.

A--> B -------------->---> E
 \-> C -------------/   /
 \-> (delay 30mins) -> D

Are there such simple linux-based tools that support this? I've looked into Airbnb's Chronos but it seems overkill for my need.

Edit: The above scenario is just a simplified version of what's happening. We have a lot more daily jobs and the dependency is a lot more complicated. So I'm actually looking for some "cron on steroid" than case-by-case bash scripts to cater each scenario.

share|improve this question

closed as off-topic by Mokubai Jul 8 at 18:05

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions seeking product, service, or learning material recommendations are off-topic because they become outdated quickly and attract opinion-based answers. Instead, describe your situation and the specific problem you're trying to solve. Share your research. Here are a few suggestions on how to properly ask this type of question." – Mokubai
If this question can be reworded to fit the rules in the help center, please edit the question.

I guess using the Jenkins CI server would be overkill as well. Still, it'd work. –  Daniel Beck May 13 '13 at 10:27
You could do this with a fairly simple bash script that runs at 00:05 (through any crond), starts job A and waits for its completion, then starts B and C in the background as well as a backgrounded subprocess which waits 30 min then starts D, then waits for all of them to finish before starting E, then when E completes simply exits. Throw in a little error handling for good measure and you should be good to go. Possibly useful links: Google "bash wait for subprocess" or similar. –  Michael Kjörling May 13 '13 at 11:36
Michael: Thanks. Though my use case is much more complicated than that. I've updated the question to reflect. –  huy May 14 '13 at 2:44
So you're looking for a simple tool to handle an unspecified more complex version of what you use as an example... Frankly, if so, I'm not sure the question as it stands is answerable. Then at least explain why the alternative you have found is "overkill" for your needs, and what specific features of it you do need. –  Michael Kjörling May 15 '13 at 13:20

3 Answers 3

As Michael Kjörling suggested in the comments, you should be able to do this with a simple bash script. Something like this:

#!/usr/bin/env bash

## Log file to which the "echo" commands bellow will write

## Change "ls /etc >/dev/null " to reflect the actual 
## jobs you want to run but keep the  "&& echo job N finished" as is.
jobA="ls /etc >/dev/null"
jobB="ls /etc >/dev/null && echo 'job B finished' >> $logfile"
jobC="ls /etc >/dev/null && echo 'job C finished' >> $logfile"
jobD="ls /etc >/dev/null && echo 'job D finished' >> $logfile"
jobE="ls /etc >/dev/null";

## Run job A, launch jobs B and C as soon as A is finished
## and launch job D 30 minutes after A finishes.
eval $jobA && (sleep 30 && eval $jobD) & eval $jobB & eval $jobC &

## Now, monitor the logfile and run job E when the rest have finished
while true; do
    lines=`wc -l $logfile | cut -f 1 -d ' '`;
    echo "$logfile : $lines"
    ## The logfile will contain 4 lines if all jobs have finished
    if [ "$lines" -eq 3 ];
      ## Run job E
      eval $jobE 
      ## Delete the logfile
      rm $logfile
      ## exit the script
      exit 0;
    ## Only check if the jobs are finished once a minute
    sleep 60;

If you use cron to launch this script at 00:05 it should do what you want. The main trick here is the use of subshels () and &&. Subshels allow you to run multiple background jobs and && to only run jobs once another job has exited successfully.

share|improve this answer
Thanks for your help @terdon. Sorry I wasn't more clear about my situation. The number of jobs I have are much more than that, and the inter dependency is a lot more complicated. The scenario I described is just a simplified version of what usually will happen. –  huy May 14 '13 at 2:39

BMC Software makes a product called Control-M which would be a perfect fit for your description of the issue. However, it isn't free :(

We use it to administer around 500 jobs in production, and somewhere close to 400 in test environments. You install clients on whatever machines you need, then set up jobs on the Control-M server to run on the clients. There are a whole lot of configurable parameters and scheduling criteria, all of which can be administered through a GUI or command line. The most fitting part to your problem is that it thrives on setting up input/output conditions for jobs so that you can have dependencies just by drag and dropping between jobs. We use it to set up workflow streams of 20+ jobs at a time.

share|improve this answer

These look interesting:

These are both python projects which aim at replacing cron with a nice GUI to see the dependencies graph.

I used to use bash for this stuff, but it's getting ugly when you grow into complex systems.

I think I will test both and give an update in case of success

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.