6

I'm using the following command to download all files off of a webpage:

wget --recursive "http://example.com"

This gives me sometimes the following error:

no-follow attribute found in www.example.com. Will not follow any links on this page

According to gnu.org, I have to add -e robots=off --wait 0.25 to my command.

My final command looks like this (I don't want span-hosts):

wget --recursive -e robots=off --wait 0.5 "http://example.com"

However I am still getting the above error. What can I do to ignore those attributes?

0

The message is a bug, wget is in fact following the links despite showing the message "Will not follow any links on this page".

This has been fixed on the master branch and should be fixed in the next version of wget.

See this for more details.

-1

In my case I had a syntax error in --follow-tags. Removing the syntax error let wget continue despite the no-follow attribute.

See more info here: https://unix.stackexchange.com/a/258850

1
  • The OP isn't using --follow-tags, though – roaima Dec 5 '20 at 17:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.