Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

For example, I want to download the dir3 of this The robot.txt exists on

It is working for,

wget -e robots=off .... 

But not working for,

wget -e robots=off ....

It is possible to download a sub directory when robots=off ?


share|improve this question
That should work. How exactly does it fail? What do you mean by "not working"? What happens? Did you check the log file (e.g. wget -o log.txt -e robots=off … – Synetech Jul 16 '12 at 16:43
--2012-07-16 23:00:14-- Resolving Connecting to||:80... connected. HTTP request sent, awaiting response... 403 Forbidden 2012-07-16 23:00:25 ERROR 403: Forbidden. – sura2k Jul 16 '12 at 17:44
The problem is, wget doesn't do DIR/ls on a directory, it can't. HTTP doesn't offer that. All it can do is follow a file like index.html which points to things. Sometimes a directory has a default index.html that points to every file in it, but often not. What do you see when you go to in a web browser? Do you see all the links to files in dir3? I doubt it. Then wget won't see them either. Maybe there is no such index.html or maybe it exists but is not readable from the web(which I guess is a common cause of 403). You can do wget -d too, it gives more info. – barlop Jul 16 '12 at 17:47
Then why wget able to download all the things including dir3 when I run it as above first example? – sura2k Jul 16 '12 at 18:07

You must log in to answer this question.

Browse other questions tagged .