15

I have a website, and I want to download all of the pages/links within that website. I want to do a wget -r on this URL. None of the links go "outside" of this specific directory, so I'm not worried about downloading the entire internet.

As it turns out, the pages I want are behind the password-protected section of the website. While I could use wget to manually do that cookie negotiation, it would be a lot easier for me to just "log in" from the browser and use some firefox plugin to recursively download everything.

Is there an extension or something that will let me do this? Many extensions focus on getting media/pictures from the page (heh. heh.) but I'm interested in all of the content - HTML and everything.

Suggestions?

Thanks!

Edit

DownThemAll seems like a cool suggestion. Can it do a recursive download? As in, download all of the links on the page, and then download all of the links contained within each of those pages, etc? So that I basically mirror the entire directory tree by following links? Kinda like the -r option of wget?

3 Answers 3

7

DownThemAll doesn't do recursive downloading. It only grabs links from the current page. Only the HTML page of the linked URLS are downloaded. The linked pages' images and media are not downloaded.

0
6

SpiderZilla is meant to do that -- but, its a bit old (2006).
It is based on HTTrack website copier.
Which has updated versions for all platforms.

There is also another older addon that can let you plug-in 'wget' itself (among other things).

However, I too feel that DownThemAll is probably a good choice.
If you know what you want to mirror, selecting the right links should not be a problem.
And, you can always tick the 'All' checkbox.

So, +1, for DownThemAll if you want to stick to the browser.
And, use HTTrack if you want a standalone tool (and wget is not handy).

Update: you may also want to look at HTTrack's votes at this bounty question,
How can I download an entire website.

1
  • DTA does not support recursion Oct 17, 2015 at 22:06
6

You can use wget -r with cookies from browser, extracted after authorization.

Firefox has "Copy as cURL" option in the context menu of the page request in the Network tab of Web Developer Tools, hotkey Ctrl+Shift+Q (you may need to reload the page after opening the tools): screenshot

Replace curl's header flag -H with wget's --header, and you have all needed headers, including cookies, to continue the browser session with wget.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .