up vote 1 down vote favorite
share [g+] share [fb]

How can i know How many and which pages, files of my site are crawl-able by search engines?

link|improve this question
45% accept rate
feedback

2 Answers

up vote 1 down vote

They are all crawlable if another page links to them. You can prevent certain web robots from indexing your pages with a robots.txt file, which they are supposed to respect, but even with one in place there are probably web robots which won't respect it. I know the big boys out there (Google, Yahoo, Bing...) will respect a robots.txt file, but there are probably smaller ones out there that may not.

link|improve this answer
is there any tool which can tell how search engine crawling my site ? – metal gear solid Jan 4 '10 at 5:50
1  
If or how ? You can check your web access logs, most robots have their own special user-agent string such as Googlebot/2.1 (+googlebot.com/bot.html) (for Google). – John T Jan 4 '10 at 5:53
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.