Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I wonder if there is a software I can use to discover all the files and sub-directories given a URL?

For example, given, I would like to find all the files in /some-directory/ directory as well as all sub-directories (and their files) in /some-directory/.

This would be for the HTTP protocol.

share|improve this question
Do you look for physical files? Or want to crawl virtual URL-s? – karatedog Dec 10 '11 at 22:35
@karatedog, i'm looking for actual files. – Mark Dec 11 '11 at 1:24
then it is easy. If directory listing is enabled on the target site (and you have proper rights), you can discover it like it was a filesystem. If directory listing is turned off, then you can't. In an Apache config that's what Indexes do. – karatedog Dec 12 '11 at 12:05
up vote 2 down vote accepted

On CMS type systems, there are no directories and subdirectories, only routes that correspond to informational nodes/IDs that are assigned to the information you are requesting. These routes are dynamically created depending on the categorization method used to access that information (newest posts, categories, tags, brand lists, and any other presentational categorization method the site owner may use to help you find the end node)

Therefore the information you are looking for may be represented by multiple variations dependent on the route used to access the end node (virtual page)

To keep the website owner happy by not overloading his server, make like Google and look for the sitemap.xml file. If the site owner is following best practice, it will be a full listing of the canonical web pages available on the website which means you only have to access the end virtual page once, not ending up downloading multiple copies of the same thing.

share|improve this answer

It depends on how the server on the site you want to crawl is set up. The URL is not always proportional to the physical directory where files are located.

Normally, if an index file is not created on a server directory, the server will return directory contents. If an index file is created, it's almost impossible to fetch directory contents directly.

However you may use a site crawler like Internet Download Manager to crawl a website by its links which are provided in a site's HTML content. IDM retrieves all the HTML/images/multimedia content/txt/PDF files on a website for you.

Be sure to check their Terms of service before crawling.

share|improve this answer
You can ask the server with a ls command ... – ragnq Dec 10 '11 at 14:57
ls is a linux command and is only available if you have root access to the server – afterburner Dec 10 '11 at 15:00
and ftp, and may work if its ssh, telnet, ... + unix – ragnq Dec 10 '11 at 15:02
@ragnq: It seems quite obvious to me that the OP doesn't own the server. – cYrus Dec 10 '11 at 15:28
@ragnq, cYrus and drfanai are right, I don't own the server so I cannot execute those commands. thanks for the input anyway. – Mark Dec 10 '11 at 15:53

wget does this, if your on *nix. Its free and open source. You can probably get it for windows too, though I'm not sure.

Of course, the limitations are the same as mentioned above. Most websites nowadays don't have URLS that map directly to directory structures, but you can effectively mirror an entire site with wget. That is, you can download all the spaces on the site which are publicly available and hyperlinked from a page that you can reach.

Many sites will block you if they detect an unauthorized crawler mirroring their site, too fast. So you may need to be polite - have the crawling program only pull down a few pages per second.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.