On CMS type systems, there are no directories and subdirectories, only routes that correspond to informational nodes/IDs that are assigned to the information you are requesting. These routes are dynamically created depending on the categorization method used to access that information (newest posts, categories, tags, brand lists, and any other presentational categorization method the site owner may use to help you find the end node)
Therefore the information you are looking for may be represented by multiple variations dependent on the route used to access the end node (virtual page)
To keep the website owner happy by not overloading his server, make like Google and look for the sitemap.xml file. If the site owner is following best practice, it will be a full listing of the canonical web pages available on the website which means you only have to access the end virtual page once, not ending up downloading multiple copies of the same thing.
Indexes
do.