I'm searching a FLOSS tool that downloads all pages (and embedded resources, e.g. images) linked in a XML sitemap (built according to http://www.sitemaps.org/). The tool should "crawl" the sitemap ...
I need to retrieve a whole website map, in a format like : http://example.org/ http://example.org/product/ http://example.org/service/ http://example.org/about/ ...
Lets say I have a sitemap.xml file with this data: <url> <loc>http://domain.com/pag1</loc> <lastmod>2012-08-25</lastmod> <changefreq>weekly</changefreq> ...
Microsoft Visio has a feature where it will create a site map for an existing website. However, my website has a login and users must be authenticated to view the content. Is there any way to enter ...
OK, currently we are restricting the directory /Assets/ from being indexed in our robots.txt. There is one directory under the /Assets/ directory that I want indexed (/Assets/product_labels/). Is ...