16

I know you can download webpages recursively using wget, but is it possible to do a dry-run? So that you could sort of do a test-run to see how much would be downloaded if you actually did it? Thinking about pages that have a lot of links to media files like for example images, audio or movie files.

3 Answers 3

24

You can use the --spider switch.

1
  • 1
    Don't forget the -nd (don't create directories) flag if you use it with the -r (recursive) flag. Jan 10, 2014 at 19:29
0

No, but you can use -R to reject media files until you're ready to actually download them.

1
  • Would you get any indication on how much you rejected?
    – Svish
    Apr 5, 2010 at 20:45
0

Using --spider as noted in paradroid's answer is great for most cases, but if you're trying to get the filename it would create (e.g. thanks to --restrict-file-names or just to reliably translate the uri's encoding), the resulting "Remote file exists" output is insufficient.

My solution was to ensure it can't write the file and then capture the error:

filename="$(wget -P /. "$uri" 2>&1 \
  |awk 'pd = index($0, ": Permission denied") {print substr($0, 4, pd - 4)}'
)"
echo "Would save: $filename"

The -P /. option specifies a directory prefix of /, root. (Oddly, wget ignores -P / and writes to the local directory instead. The dot works around that.)

Note that you should not run this as a user with write access to /. If you have such access, try this instead:

filename="$(
  d="$(mktemp -d)";
  chmod 000 $d;
  wget -P $d "$uri" 2>&1 |awk -v d=$d/ '
    a = index($0, d) && b = index($0, ": Permission denied") {
      a += length(d);
      print substr($0, a, b - a)
    }
  ';
  rm -rf $d
)"
echo "Would save: $filename"

This makes a temporary directory, renders it unwritable, then runs wget with it as the prefix. It then removes the temporary directory.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .