Tell me more ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.
up vote 0 down vote favorite

For example, I want to download the dir3 of this http://example.com/dir1/dir2/dir3. The robot.txt exists on http://example.com/dir1/

It is working for,

wget -e robots=off .... http://example.com/dir1/

But not working for,

wget -e robots=off .... http://example.com/dir1/dir2/dir3

It is possible to download a sub directory when robots=off ?

Thanks!

share|improve this question
That should work. How exactly does it fail? What do you mean by "not working"? What happens? Did you check the log file (e.g. wget -o log.txt -e robots=off … http://example.com/dir1/dir2/dir3/)? – Synetech Jul 16 '12 at 16:43
--2012-07-16 23:00:14-- example.com/dir1/dir2/dir3 Resolving example.com... xxx.xxx.xxx.xxx Connecting to example.com|xxx.xxx.xxx.xxx|:80... connected. HTTP request sent, awaiting response... 403 Forbidden 2012-07-16 23:00:25 ERROR 403: Forbidden. – sura2k Jul 16 '12 at 17:44
The problem is, wget doesn't do DIR/ls on a directory, it can't. HTTP doesn't offer that. All it can do is follow a file like index.html which points to things. Sometimes a directory has a default index.html that points to every file in it, but often not. What do you see when you go to example.com/dir1/dir2/dir3 in a web browser? Do you see all the links to files in dir3? I doubt it. Then wget won't see them either. Maybe there is no such index.html or maybe it exists but is not readable from the web(which I guess is a common cause of 403). You can do wget -d too, it gives more info. – barlop Jul 16 '12 at 17:47
Then why wget able to download all the things including dir3 when I run it as above first example? – sura2k Jul 16 '12 at 18:07

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.