1

I have a website filesystem wherein each file has two copies, a summary copy and a large copy, with different names. For example, a large copy file might be pages/info/GK2grkG23r_lg.html while in the summary copy the letters pre-extension are changed as pages/info/GK2grkG23r_sh.html. I'm trying to run a wget on the file system, but all the links by design point to the small copy to save resources, and I need the large copies. So while running wget over pages might download all the summary copies:

pages/info/GK2grkG23r_sh.html
pages/info/2Yp58xr4Ir_sh.html
pages/info/Lmk2nIj23r_sh.html
pages/info/M23Dka6D4r_sh.html

I'd like to change the names of the files wget is going to download to:

pages/info/GK2grkG23r_lg.html
pages/info/2Yp58xr4Ir_lg.html
pages/info/Lmk2nIj23r_lg.html
pages/info/M23Dka6D4r_lg.html

Even though the files wget will find will point to the first set.

To be clear, I do not want to download the files with the first set of names above and then rename them. I want to run wget, which will then try to download the first list of files, but change the letters slightly so it downloads the second list. Can wget do this?

1

You can try with a workaround.

The milestons are:

  1. Download the full list of files and store it in file
  2. Change the final part of the filenames
  3. Download only the files with modified name.

You need a file filter.awk similar to this

/^--.*--  http:\/\/.*[^\/]$/ { u=$3; }
/^Length: [[:digit:]]+/ { print u; }

Step by step:

wget -r -np --spider http://yourhost | awk -f filter.awk  > My_List.txt
sed -i s/_sh.html/_lg.html/g My_List.txt
wget -x -i My_List.txt

Credits:
I take inspiration from this answer.

  • I like it, and as a plus you end up with a convenient index of your files afterward – Philip Arnott Sep 2 '14 at 18:29
  • Warning it's far to be perfect... there are particular cases that are not managed by the script (as you can see from the note of the inspiration). Moreover you can modify > My_List.txt with | sort -u > My_List.txt in case your files are called from more than one page in the site: else, in this case, you download them more than one time, losing time, resources... It was a sketch more than an answer :) – Hastur Sep 3 '14 at 7:13
0

First, get the index file:

wget -O index.html "$URI"

Then, rewrite the URLs in it:

sed -i '/_sh\(\.html\)/_lg\1/g' index.html

(I used sed, but you might prefer, e.g. an XSLT-based approach. Take your pick.)

Now, tell wget to grab all the pages/info/*, using the modified index file and with its original location as the starting base URI:

wget --recursive --level=1 \
    --input-file=index.html --force-html --base="$URI" \
    --accept-regex 'pages/info/.*_lg\.html'

N.B. most of those long options have short alternatives - see the manpage. I used the long form for clarity.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.