I'm maintaining a data-warehousing system that involves a lot of dependent jobs (data import, transform, etc). I've been using Linux's
crontab to manage them until the dependency between jobs get complicated.
Basically I'm looking for some
cron replacement that helps me with the following scenario:
Run job A at
00:05(easy). Usually this is the import job.
Schedule job B, C, D to run after job A finishes. Job D only runs 30 mins after job A finishes (to distribute load). These are the transform jobs.
Job E run when all B, C, D finish. Usually this is the job that bring aggregated data to a web front-end database.
All of these happen on the same node.
I imagine it looks like a topology graph.
A--> B -------------->---> E \-> C -------------/ / \-> (delay 30mins) -> D
Are there such simple linux-based tools that support this? I've looked into Airbnb's Chronos but it seems overkill for my need.
Edit: The above scenario is just a simplified version of what's happening. We have a lot more daily jobs and the dependency is a lot more complicated. So I'm actually looking for some "cron on steroid" than case-by-case bash scripts to cater each scenario.