Questions tagged [sitemap]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
1 answer
654 views

Download/update webpages listed in XML sitemap [closed]

I'm searching a FLOSS tool that downloads all pages (and embedded resources, e.g. images) linked in a XML sitemap (built according to http://www.sitemaps.org/). The tool should "crawl" the sitemap ...
unor's user avatar
  • 3,096
3 votes
5 answers
16k views

Website crawler/spider to get site map [closed]

I need to retrieve a whole website map, in a format like : http://example.org/ http://example.org/product/ http://example.org/service/ http://example.org/about/ http://example.org/product/viewproduct/...
ack__'s user avatar
  • 117
6 votes
6 answers
15k views

Extract Links from a sitemap(xml)

Lets say I have a sitemap.xml file with this data: <url> <loc>http://domain.com/pag1</loc> <lastmod>2012-08-25</lastmod> <changefreq>weekly</changefreq> <...
Akshat Mittal's user avatar
1 vote
2 answers
4k views

How can I create a Site Map for a Password Protected Site

Microsoft Visio has a feature where it will create a site map for an existing website. However, my website has a login and users must be authenticated to view the content. Is there any way to enter ...
Bob's user avatar
  • 11
2 votes
2 answers
413 views

Unrestricting a sub directory in robots.txt

OK, currently we are restricting the directory /Assets/ from being indexed in our robots.txt. There is one directory under the /Assets/ directory that I want indexed (/Assets/product_labels/). Is ...
xxl3ww's user avatar
  • 1,499