Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

How can i know How many and which pages, files of my site are crawl-able by search engines?

share|improve this question

They are all crawlable if another page links to them. You can prevent certain web robots from indexing your pages with a robots.txt file, which they are supposed to respect, but even with one in place there are probably web robots which won't respect it. I know the big boys out there (Google, Yahoo, Bing...) will respect a robots.txt file, but there are probably smaller ones out there that may not.

share|improve this answer
is there any tool which can tell how search engine crawling my site ? – metal gear solid Jan 4 '10 at 5:50
If or how ? You can check your web access logs, most robots have their own special user-agent string such as Googlebot/2.1 ( (for Google). – John T Jan 4 '10 at 5:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .