I have come to the end of my research with unix:> I'm new with unix and its enviroment, so i need your help guys. Long story short, i have a housekeeping script, which runs daily/weekly (it doesnt matter) and it gzip's loads of files total of ~20Tb. The old script is using cycle and it gzips files one by one, until it matches the pattern in config file. What I want to do is to make gzip files 4 at once. Since it will reduce the time of housekeeping script. With my research ive come to conclusion: find ${var} ! -name "${name}" -type f 2>/dev/null | xargs -n 1 -P 4 gzip

My question is, is there any way to make this happen without find/ls commands also can i substitute xargs with some different commands? Please note that i cannot install any aditional packages.

  • Normally I'd recommend looking at GNU Parallel for such tasks. Do you have it installed, or does it fall in the "cannot install additional packages" category. – Dmitry Grigoryev Jun 15 '15 at 12:14
  • Sadly its on that category ;( – Justas Jun 15 '15 at 12:14

I think the issue is that your script matches files and start to zip, matches the next and starts to zip. My advice:

Split your script up in order to be able to run multiple instances of gzip, as gzip cant be multithreaded (You can however run multiple instances of it at the same time to get all the cores going)

I referred to this post for the code required to start multiple instances.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.