Wget
The by far easiest solution would be installing Wget and executing the following command:
wget --input-file=gwurls.txt
Wget automatically renames the output file if a file of the same name already exists.
Renaming
If you strip scheme and host (e.g., http://example.com/
) from the URL, you can replace all slashes with underscores (or any other character) and save the files like that. To be on the safe side, you could replace pre-exisiting underscores with double underscores.
With bash, this should work:
while read -r URL; do
OUTPUT="${URL#http://example.com/}"
OUTPUT="${OUTPUT//_/__}"
OUTPUT="${OUTPUT//\//_}"
curl --output "$OUTPUT" --url "$URL"
done < gwurls.txt
How it works:
while read -r URL; do ... done < gwurls.txt
reads the contents of gwurls.txt line by line and stores the entire line (without leading or trailing spaces) into the variable URL and executes ...
.
The three OUTPUT=...
commands perform the mentioned replacements using bash string manipulation.
curl --output "$OUTPUT" --url "URL"
downloads the file and stores it with the desired filename.
Directory structure
It's also possible to re-create the directory structure of the server using a similar approach.
With bash, this should work:
while read -r URL; do
OUTPUT="${URL#http://example.com/}"
curl --create-dirs --output "$OUTPUT" --url "$URL"
done < gwurls.txt
Here, the --create-dirs
switch makes cURL create the directory a
if OUTPUT reads a/1.pdf
.