I'm downloading all the files from a website below a specific node.

wget \
    --recursive \
    --no-clobber \
    --page-requisites \
    --html-extension \
    --convert-links \
    --domains www.___.uk \
    --no-parent \

When wget finds a link to a file is it possible to tell wget to also download the index listing of the parent directory (and grandparent, etc) and all other files linked to from those pages?

For example:

download /index.html (1 new link found)
download /pdfs/foo.pdf
download /pdfs/ (1 new link found)
download /pdfs/bar.pdf
  • --recursive and --level might help, but since you've specified --no-parent it won't ascend to the parent directory. – Karan Dec 12 '12 at 2:36
  • @Karan I don't want to go above the initial starting node, but I do want to get parents of grand-children. --no-parent seems to stop both of those happening. --level only seems to stop it descending further? – Annan Dec 12 '12 at 3:16

the way you've described what your want, your solution, looks like you're thinking of an algorithm, you won't get that from wget I don't think. What it can do and the main thing it does, is look at a webpage, download any files linked to on it, and any webpages linked to from there and any files that those webpages link to.

Considering your conditional of downloading a (parent) page, IF/on condition that, there's a file.. the fact is, putting aside the conditional aspect wget won't necessarily even be able to go to - download- a parent page given a webpage to download, because it doesn't work like that. It only sees what is linked to. Just like it can only download files it sees links to.

if the webpages in each directory are the type of directory index pages that include . and .. then wget will be able to download a webpage and recurse backwards up the directory tree. (it won't go as far as foreign hosts unless you do -H), but I suppose if you don't want to go all the way up downloading the whole website, then you'd want to use -np and then it wouldn't be able to go back/up the directory tree at all, above the address of the webpage you specify to wget.

let's pretend for a moment that wget could go backwards (and to an extent it can as mentioned in above paragraph), first I don't think you can say "only download the parent directory if there's a file". But that aside. I don't think it matters if you download a webpage that doesn't have a file. The webpage, the htm/html file, is just some markup and some links, it doesn't take up much space.

What you could do, is download the whole website, as much as wget will allow (wget cannot do ls/dir on a directory and see all that is there. it only knows of what is linked to, and is given a webpage to start with). You can download the lot. And then work on fixing your local copy. deleting unwanted directories, unwanted webpages, etc maybe automating some of that.

I think your question at the moment, is confused.. as to how wget operates. I hope this helps and maybe you can rethink what you want wget to do.

If you wanted wget to download those 2 files, you'd need a webpage linking to them either directly, or indirectly, by pointing to another webpage that links to them. wget doesn't really go backwards or forwards. It doesn't work like that. But it can be told not to go back beyond a certain point. And while the default is not to go to foreign hosts recursively, it can with -H be told that it can go to foreign hosts recursively.

Maybe Curl can do things wget can't that are relevant to what you want, but I don't know. I doubt wget can. but consider other solutions within how wget works, which i've described somewhat.

If you download a whole site for example, or not even the whole site, but just more than you want, You will have more power locally, as you can then do ls/dir on what you have, get nice lists, delete what you don't want, automate. You may also want to rethink what you'd want to ask wget to do.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.