Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a list of files on a server, let's say matching the pattern http://example.com/abcd-{0..99}.zip.

I want to download, then decompress all these files (they're just zipped text, with one file in each .zip), concatenate them into a single stream, then pipe it into a different program. I know to download them, it's best to use curl, and to unzip, I can use funzip, but funzip only takes a single .zip as a parameter.

Is bash capable of doing this painlessly (if so, how?), or should I just give up and do this in python?

share|improve this question
    
I am not sure about python, but you can do it using bash ( as you said using curl and unzip ). – user122530 May 17 '12 at 3:07

You didn't say why you wanted to use curl and funzip. I think for your purposes wget and unzip would work better. Say your list of files is in the file filelist. Then to download all those files, execute

wget -i filelist

Then to unzip those files and pipe them into some program, execute

unzip -p 'abcd-*.zip' | program

Note that you must quote the zip file name pattern given to unzip so that it is not expanded by your shell.

If you don't already have a list of files to be downloaded, but their names follow the pattern you gave, you can create a list as follows.

seq 0 99 | while read n; do echo "http://example.com/abcd-$n.zip"; done > filelist
share|improve this answer
    
printf "http://example.com/abcd-%d.zip\n" {0..99} – grawity May 17 '12 at 11:08
up vote 0 down vote accepted

The script I ended up writing to do this is:

echo http://example.com/some-data{0..799}.csv.zip | xargs -n1 -P8 ./download-file

and the contents of download-file are:

#!/bin/bash
NAME=$(basename $1 .zip).xz
wget -O- $1 | funzip | nice -n5 xz -z9 > $NAME
share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .