33

I am trying to download a full website directory using CURL. The following command does not work:

curl -LO http://example.com/

It returns an error: curl: Remote file name has no length!.

But when I do this: curl -LO http://example.com/someFile.type it works. Any idea how to download all files in the specified directory? Thanks.

31

HTTP doesn't really have a notion of directories. The slashes other than the first three (http://example.com/) do not have any special meaning except with respect to .. in relative URLs. So unless the server follows a particular format, there's no way to “download all files in the specified directory”.

If you want to download the whole site, your best bet is to traverse all the links in the main page recursively. Curl can't do it, but wget can. This will work if the website is not too dynamic (in particular, wget won't see links that are constructed by Javascript code). Start with wget -r http://example.com/, and look under “Recursive Retrieval Options” and “Recursive Accept/Reject Options” in the wget manual for more relevant options (recursion depth, exclusion lists, etc).

If the website tries to block automated downloads, you may need to change the user agent string (-U Mozilla), and to ignore robots.txt (create an empty file example.com/robots.txt and use the -nc option so that wget doesn't try to download it from the server).

  • How wget is able to do it. ?? – Srikan Oct 6 '16 at 16:29
  • @Srikan wget parses the HTML to find the links that it contains and recursively downloads (a selection of) those links. – Gilles Oct 6 '16 at 21:05
  • If the files don't have any internal links, then does recursive download fail to get all the files. Lets say there is a HTTP folder of some txt files. Will wget succeed to get all the files. Let me try it after this comment – Srikan Oct 15 '16 at 2:28
  • @Srikan HTTP has no concept of directory. Recursive download means following links in web pages (including web pages generated by the server to show a directory listing, if the web server does this). – Gilles Oct 15 '16 at 11:58
  • wget supports ignoring robots.txt with the flag -e robots=off. Alternatively you can avoid downloading it by rejecting it with -R "robots.txt". – Ryan Krage Nov 13 '18 at 13:39
26

Always works for me, included no parent and recursive to only get the desired directory.

 wget --no-parent -r http://WEBSITE.com/DIRECTORY
  • This should be the accepted answer. – Juan Jimenez Apr 19 at 12:12
13

In this case, curl is NOT the best tool. You can use wget with the -r argument, like this:

wget -r http://example.com/ 

This is the most basic form, and and you can use additional arguments as well. For more information, see the manpage (man wget).

5

This isn't possible. There is no standard, generally implemented, way for a web server to return the contents of a directory to you. Most servers do generate an HTML index of a directory, if configured to do so, but this output isn't standard, nor guaranteed by any means. You could parse this HTML, but keep in mind that the format will change from server to server, and won't always be enabled.

  • Look at this app called Site Sucker. sitesucker.us. How do they do it? – Foo Oct 17 '10 at 18:09
  • They parse the HTML file and download every link in it. – Brad Oct 17 '10 at 18:14
  • Using wget or curl? – Foo Oct 17 '10 at 18:17
  • 7
    @Brad: curl doesn't parse the HTML, but wget does precisely this (it's called recursive retrieval). – Gilles Oct 17 '10 at 20:00
  • 1
    Ah, well I stand corrected! gnu.org/software/wget/manual/html_node/… OP should be aware that this still doesn't get what he is looking for... it only follows links that are available on the pages returned. – Brad Oct 17 '10 at 20:13
2

You can use the Firefox extension DownThemAll! It will let you download all the files in a directory in one click. It is also customizable and you can specify what file types to download. This is the easiest way I have found.

0

You might find a use for a website ripper here, this will download everything and modify the contents/internal links for local use. A good one can be found here: http://www.httrack.com

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.