I'm using the following command to download all files off of a webpage:
wget --recursive "http://example.com"
This gives me sometimes the following error:
no-follow attribute found in www.example.com. Will not follow any links on this page
According to gnu.org, I have to add
-e robots=off --wait 0.25 to my command.
My final command looks like this (I don't want
wget --recursive -e robots=off --wait 0.5 "http://example.com"
However I am still getting the above error. What can I do to ignore those attributes?