Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I am trying to download a full website directory using CURL. The following command does not work:

curl -LO http://example.com/

It returns an error: curl: Remote file name has no length!.

But when I do this: curl -LO http://example.com/someFile.type it works. Any idea how to download all files in the specified directory? Thanks.

share|improve this question

6 Answers 6

HTTP doesn't really have a notion of directories. The slashes other than the first three (http://example.com/) do not have any special meaning except with respect to .. in relative URLs. So unless the server follows a particular format, there's no way to “download all files in the specified directory”.

If you want to download the whole site, your best bet is to traverse all the links in the main page recursively. Curl can't do it, but wget can. This will work if the website is not too dynamic (in particular, wget won't see links that are constructed by Javascript code). Start with wget -r http://example.com/, and look under “Recursive Retrieval Options” and “Recursive Accept/Reject Options” in the wget manual for more relevant options (recursion depth, exclusion lists, etc).

If the website tries to block automated downloads, you may need to change the user agent string (-U Mozilla), and to ignore robots.txt (create an empty file example.com/robots.txt and use the -nc option so that wget doesn't try to download it from the server).

share|improve this answer

This isn't possible. There is no standard, generally implemented, way for a web server to return the contents of a directory to you. Most servers do generate an HTML index of a directory, if configured to do so, but this output isn't standard, nor guaranteed by any means. You could parse this HTML, but keep in mind that the format will change from server to server, and won't always be enabled.

share|improve this answer
    
Look at this app called Site Sucker. sitesucker.us. How do they do it? –  Foo Oct 17 '10 at 18:09
    
They parse the HTML file and download every link in it. –  Brad Oct 17 '10 at 18:14
    
Using wget or curl? –  Foo Oct 17 '10 at 18:17
4  
@Brad: curl doesn't parse the HTML, but wget does precisely this (it's called recursive retrieval). –  Gilles Oct 17 '10 at 20:00
1  
Ah, well I stand corrected! gnu.org/software/wget/manual/html_node/… OP should be aware that this still doesn't get what he is looking for... it only follows links that are available on the pages returned. –  Brad Oct 17 '10 at 20:13

Always works for me, included no parent and recursive to only get the desired directory.

 wget --no-parent -r http://WEBSITE.com/DIRECTORY
share|improve this answer

In this case, curl is NOT the best tool. You can use wget with the -r argument, like this:

wget -r http://example.com/ 

This is the most basic form, and and you can use additional arguments as well. For more information, see the manpage (man wget).

share|improve this answer

You can use the Firefox extension DownThemAll! It will let you download all the files in a directory in one click. It is also customizable and you can specify what file types to download. This is the easiest way I have found.

share|improve this answer

You might find a use for a website ripper here, this will download everything and modify the contents/internal links for local use. A good one can be found here: http://www.httrack.com

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.