Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

@slhck provided a useful answer to this similar question.

Downloading multiple files with wget and handling parameters

This code was provided to read urls from a text file filelist, then run a loop to download each.

    #!/usr/bin/env bash 
    while read line 
        wget -c --load-cookies cookies.txt $line -O ${line##*/} 
    done < filelist


As I'm not familiar with shell scripts, I am wondering if you could download multiple xml files by:

  • creating a simple comma seperated text file filelist
  • read from:


renamedfile.xml, url-01.php
renamedFileDifferent.xml, url-02.php
specificFileRename.xml, "url-03"
newFilename.xml, "url-04"

  • read through each line
  • split the line into newfile, remoteFile
  • And run:  

    wget -O newfile remoteFile >/dev/null 2>&1 


Is it possible to write this in a shell script? Can you provide an example?

share|improve this question
up vote 1 down vote accepted

To get the text from before the comma you use ${line/,*}.

(What this actually does is replace ",*" or all the text from the comma onwards with empty string - leaving only the part of the string before the comma)

To get the text from after the comma you use ${line/*,}.

So the full command would be:

while read line
    wget -O ${line/,*} ${line/*,} >/dev/null 2>&1 
done < filelist

or, on one line:

while read line; do echo wget -O ${line/,*} ${line/*,} >/dev/null 2>&1; done < filelist

In Windows (assuming you've installed wget, from, and set your path correctly), it would be:

for /f "delims=, tokens=1*" %a in (filelist) do wget -O %a %b
share|improve this answer
What directory is a good place to put the text file, and shell file typically? This will be run as a cron job. – rrrfusco Sep 25 '12 at 0:59
Wherever you want to run it from (and save the files to). eg. if you put filelist and in sub directory of your home, and then run it from there, that's where it will save the files. – Luke Sep 25 '12 at 1:03
I'm not sure it will run as expected if run as a cron job. You will want to specify absolute paths in the filelist (~\downloaddir\renamedfile.xml) and probably in the crontab as well (46 8 * * cd ~\downloaddir; or maybe just 46 8 * * ~\downloaddir\ Refer… and… – Luke Sep 25 '12 at 11:09
another option is to harcode the paths into the script, eg. wget -O ~/downloaddir/${line/,*} ${line/*,} >/dev/null 2>&1 and done < ~/downloaddir/filelist – Luke Sep 25 '12 at 11:12

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.