I googled for this but couldn't find an example of how to use eg. wget to download the directory structure of a website without downloading any of its files.

At this point, I just need to understand how a website is organized. I'll mirror the site later.

  • Do you mean the directory structure as in where the images, css, and js are stored? – Paul Apr 6 '16 at 9:48
  • Yes, the directory structure, eg. /dir1, /dir2, etc. – OverTheRainbow Apr 6 '16 at 9:55
  • This isn't clear. Do you mean the directory structure of the assets that make up the site, such as where the images and css etc files reside, or do you mean the tree structure that describes how the pages link together (which is not related to directory structure). What for example would be in "dir1" in this case? – Paul Apr 6 '16 at 13:28
  • This looks very relevant to web crawl the list of web pages on a particular web site. stackoverflow.com/questions/857653/… – simpleb Apr 6 '16 at 14:32

At the command prompt type:

wget -r --spider www.your-website.com

Alternate command with an option to specify the maximum depth level depth.

wget -r --spider -l depth www.your-website.com

Recursive retrieval options:

    Turn on recursive retrieving.    The default maximum depth is 5.

    Don't download anything

-l depth
    Specify recursion maximum depth level depth.
  • but this does download the files, too. – xtofl Oct 8 '18 at 7:52
  • The --spider option of wget doesn't download anything except for creating the empty directory structure of the URL without downloading any files into the directories. – karel Oct 8 '18 at 9:50
  • Somehow, I read over that argument. My bad. – xtofl Oct 9 '18 at 6:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.