I know you can download webpages recursively using wget
, but is it possible to do a dry-run? So that you could sort of do a test-run to see how much would be downloaded if you actually did it? Thinking about pages that have a lot of links to media files like for example images, audio or movie files.
3 Answers
You can use the --spider
switch.
-
1Don't forget the
-nd
(don't create directories) flag if you use it with the-r
(recursive) flag. Jan 10, 2014 at 19:29
No, but you can use -R
to reject media files until you're ready to actually download them.
Using --spider
as noted in paradroid's answer is great for most cases, but if you're trying to get the filename it would create (e.g. thanks to --restrict-file-names
or just to reliably translate the uri's encoding), the resulting "Remote file exists" output is insufficient.
My solution was to ensure it can't write the file and then capture the error:
filename="$(wget -P /. "$uri" 2>&1 \
|awk 'pd = index($0, ": Permission denied") {print substr($0, 4, pd - 4)}'
)"
echo "Would save: $filename"
The -P /.
option specifies a directory prefix of /
, root. (Oddly, wget
ignores -P /
and writes to the local directory instead. The dot works around that.)
Note that you should not run this as a user with write access to /
. If you have such access, try this instead:
filename="$(
d="$(mktemp -d)";
chmod 000 $d;
wget -P $d "$uri" 2>&1 |awk -v d=$d/ '
a = index($0, d) && b = index($0, ": Permission denied") {
a += length(d);
print substr($0, a, b - a)
}
';
rm -rf $d
)"
echo "Would save: $filename"
This makes a temporary directory, renders it unwritable, then runs wget
with it as the prefix. It then removes the temporary directory.