0

I have a Jenkins build server (Linux) and an agent (Windows) that both create large temporary workspaces that can be effectively treated like caches:

  • it would be nice if they were kept
  • they can be restored if lost
  • if lost, data should be missing instead of corrupted

Is there a file system that allows me to sacrifice persistence of data for speed, still allows me to keep data across reboots, but discards data potentially affected by unclean shutdowns, ideally with a granularity of "toplevel directory"?

A typical scenario would be that this would be mounted to the directory containing the workspaces, so each job creates a toplevel directory. While the job is running, various files deep in the hierarchy are read from and written to. If the machine crashes while a job is running, and files might be in an inconsistent state, the entire job directory should be removed during the file system check, because it can be easily recreated.

1

Don't think you'll find what you want in a file system.

What you could do though is as the last step of the job (at a point you'd want to save the cache) have it create a file with a consistent name - /storage/jobID/job_complete or something. On start up, if that file doesn't exist in a job directory, rm -r the job directory. That solves your "half finished job" issue.

Next, you want speed. Time to look at either new hardware like SSDs or consider implementing RAID 0. While the 0 indicates how much data you can recover if one drive dies, you can get a big speed improvement for both writes and reads. (Media center I supported at a local college uses RAID-1 for live video capture before moving the resulting files to SAN storage).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.