How can i know How many and which pages, files of my site are crawl-able by search engines?
|
feedback
|
|
They are all crawlable if another page links to them. You can prevent certain web robots from indexing your pages with a robots.txt file, which they are supposed to respect, but even with one in place there are probably web robots which won't respect it. I know the big boys out there (Google, Yahoo, Bing...) will respect a robots.txt file, but there are probably smaller ones out there that may not. | |||||||
feedback
|
|
I found my answer here http://www.seobench.com/search-engine-crawler-simulator/index.php | |||
|
feedback
|