1

I'm trying to save a local version of the clojure docs with the wget command:

wget --user-agent=firefox --ignore-length -c -km 
"http://clojuredocs.org/quickref/Clojure Core"`

but keep getting a no such file or directory when it reaches some links like http://clojuredocs.org/clojure_core/clojure.core/rem which obviously exist because you can get there with a browser. I'm guessing this is a problem with the way wget is building/concatenating the path. How do you fix this?. I've tried other options like --user-agent and --ignore-length but I keep getting the same results as seen below(you'll have to open the image to see the messages properly).

enter image description here

This seems to be a problem with mirroring because the command:

wget http://clojuredocs.org/clojure_core/clojure.core/rem

works okay.

  • http://clojuredocs.org/clojure_core get's saved as a file named clojure_core. That's why no folder clojure_core can be created. Perhaps you can work around with -nd which doesn't create a directory hierarchy. – mpy Nov 10 '13 at 13:18
  • Yeah, I see that. It works when I create the folder clojure_core first so that any other file named clojure_core will be renamed to clojure_core1 and so on. Question is, why does it do that? I've tried the -nd option and it just causes other problems, e.g it gets stuck in a loop trying to download robots.txt so many times. – gebby Nov 10 '13 at 16:06
3

Your problem originates from the behavior of wget to save the URL http://clojuredocs.org/clojure_core to a file named ./clojuredocs.org/clojure_core, but the URL http://clojuredocs.org/clojure_core/ (notice the trailing slash) to a file named ./clojuredocs.org/clojure_core/index.html.

Once the file ./clojuredocs.org/clojure_core is created, following downloads of e.g. http://clojuredocs.org/clojure_core/something are doomed to fail, because wget can't create a directory ./clojuredocs.org/clojure_core anymore.

This was reported as bug #29647 on the GNU Wget Bugtracker.

With the provided patch (which obviously didn't make into the official source code) this problem vanishes and wget is forced to create the directory first. So, the download can continue.

However, http://clojuredocs.org/clojure_core gets saved as ./clojuredocs.org/clojure_core.1, not as ./clojuredocs.org/clojure_core/index.html.

I cannot judge if the link-converter (-k) is smart enough to make the links in this mirrored local copy working... I stopped the download after a few minutes. (I'm too impatient ;))

  • I had no luck with that patch. It introduces undefined methods that break the build. That site has major issues, you knew right canceling the download. It needs a lot of baby sitting and manual intervention. I gave up too after about 20mb. I'm going to have to work with that. Thanks for the answer, I wouldn't have looked in the bugtracker. – gebby Nov 11 '13 at 10:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.