The tag has no wiki summary.

learn more… | top users | synonyms

0
votes
1answer
69 views

Download/update webpages listed in XML sitemap

I'm searching a FLOSS tool that downloads all pages (and embedded resources, e.g. images) linked in a XML sitemap (built according to http://www.sitemaps.org/). The tool should "crawl" the sitemap ...
0
votes
5answers
4k views

Website crawler/spider to get site map [closed]

I need to retrieve a whole website map, in a format like : http://example.org/ http://example.org/product/ http://example.org/service/ http://example.org/about/ ...
1
vote
3answers
2k views

Extract Links from a sitemap(xml)

Lets say I have a sitemap.xml file with this data: <url> <loc>http://domain.com/pag1</loc> <lastmod>2012-08-25</lastmod> <changefreq>weekly</changefreq> ...
0
votes
2answers
1k views

How can I create a Site Map for a Password Protected Site

Microsoft Visio has a feature where it will create a site map for an existing website. However, my website has a login and users must be authenticated to view the content. Is there any way to enter ...
2
votes
2answers
97 views

Unrestricting a sub directory in robots.txt

OK, currently we are restricting the directory /Assets/ from being indexed in our robots.txt. There is one directory under the /Assets/ directory that I want indexed (/Assets/product_labels/). Is ...