Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a list of files on a server, let's say matching the pattern http://example.com/abcd-{0..99}.zip.

I want to download, then decompress all these files (they're just zipped text, with one file in each .zip), concatenate them into a single stream, then pipe it into a different program. I know to download them, it's best to use curl, and to unzip, I can use funzip, but funzip only takes a single .zip as a parameter.

Is bash capable of doing this painlessly (if so, how?), or should I just give up and do this in python?

share|improve this question
    
I am not sure about python, but you can do it using bash ( as you said using curl and unzip ). –  user122530 May 17 '12 at 3:07

2 Answers 2

You didn't say why you wanted to use curl and funzip. I think for your purposes wget and unzip would work better. Say your list of files is in the file filelist. Then to download all those files, execute

wget -i filelist

Then to unzip those files and pipe them into some program, execute

unzip -p 'abcd-*.zip' | program

Note that you must quote the zip file name pattern given to unzip so that it is not expanded by your shell.

If you don't already have a list of files to be downloaded, but their names follow the pattern you gave, you can create a list as follows.

seq 0 99 | while read n; do echo "http://example.com/abcd-$n.zip"; done > filelist
share|improve this answer
    
printf "http://example.com/abcd-%d.zip\n" {0..99} –  grawity May 17 '12 at 11:08
up vote 0 down vote accepted

The script I ended up writing to do this is:

echo http://example.com/some-data{0..799}.csv.zip | xargs -n1 -P8 ./download-file

and the contents of download-file are:

#!/bin/bash
NAME=$(basename $1 .zip).xz
wget -O- $1 | funzip | nice -n5 xz -z9 > $NAME
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.