Questions tagged [sitemap]
The sitemap tag has no usage guidance.
5
questions
0
votes
1
answer
654
views
Download/update webpages listed in XML sitemap [closed]
I'm searching a FLOSS tool that downloads all pages (and embedded resources, e.g. images) linked in a XML sitemap (built according to http://www.sitemaps.org/).
The tool should "crawl" the sitemap ...
3
votes
5
answers
16k
views
Website crawler/spider to get site map [closed]
I need to retrieve a whole website map, in a format like :
http://example.org/
http://example.org/product/
http://example.org/service/
http://example.org/about/
http://example.org/product/viewproduct/...
6
votes
6
answers
15k
views
Extract Links from a sitemap(xml)
Lets say I have a sitemap.xml file with this data:
<url>
<loc>http://domain.com/pag1</loc>
<lastmod>2012-08-25</lastmod>
<changefreq>weekly</changefreq>
<...
1
vote
2
answers
4k
views
How can I create a Site Map for a Password Protected Site
Microsoft Visio has a feature where it will create a site map for an existing website. However, my website has a login and users must be authenticated to view the content.
Is there any way to enter ...
2
votes
2
answers
413
views
Unrestricting a sub directory in robots.txt
OK, currently we are restricting the directory /Assets/ from being indexed in our robots.txt. There is one directory under the /Assets/ directory that I want indexed (/Assets/product_labels/). Is ...