Sign up ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

For example, I want to download the dir3 of this The robot.txt exists on

It is working for,

wget -e robots=off .... 

But not working for,

wget -e robots=off ....

It is possible to download a sub directory when robots=off ?


share|improve this question
That should work. How exactly does it fail? What do you mean by "not working"? What happens? Did you check the log file (e.g. wget -o log.txt -e robots=off … – Synetech Jul 16 '12 at 16:43
--2012-07-16 23:00:14-- Resolving Connecting to||:80... connected. HTTP request sent, awaiting response... 403 Forbidden 2012-07-16 23:00:25 ERROR 403: Forbidden. – sura2k Jul 16 '12 at 17:44
The problem is, wget doesn't do DIR/ls on a directory, it can't. HTTP doesn't offer that. All it can do is follow a file like index.html which points to things. Sometimes a directory has a default index.html that points to every file in it, but often not. What do you see when you go to in a web browser? Do you see all the links to files in dir3? I doubt it. Then wget won't see them either. Maybe there is no such index.html or maybe it exists but is not readable from the web(which I guess is a common cause of 403). You can do wget -d too, it gives more info. – barlop Jul 16 '12 at 17:47
Then why wget able to download all the things including dir3 when I run it as above first example? – sura2k Jul 16 '12 at 18:07

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.