Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Is there any site which can be rejected by a crawler? I am using the Burp Suite crawler for the time being to crawl the sites.

I want to know when and in which cases a crawler fails to retrieve the results, as I have to build such a site which should reject crawler requests.

I have been running the above mentioned crawler on random sites, but couldn't find any specific site which rejected the crawler requests. Somehow, Burp Suite managed to get all the data from the sites.

Is this possible? Which sites reject these crawler requests?

share|improve this question

Are you looking for robots.txt?

Just put this in a file called robots.txt in your webroot and the site will not be crawled

User-agent: *
Disallow: /

There are many ways to tweak the behavior, visit to learn more.

share|improve this answer
That won't work for the "Burp Site crawler" mentioned in the question, which works in conjunction with actual human beings browsing the site and scrapes its data passively from what they see. Read its documentation, hyperlinked-to in the question, for details. – JdeBP Oct 6 '11 at 9:44

Nifle mentions robots.txt. That's a coöperative mechanism for WWW spiders to recognize sites that don't want to be crawled. Actually rejecting the crawler is a slightly different thing, that from a wide perspective few WWW sites do (since robots.txt exists).

It is done by the content HTTP server for the site "knowing" the IP addresses, User-Agent: headers, or other information associated with a known WWW spider, and behaving differently when it recognizes a spider requesting a page/file/image.

There are several major variations on this theme:

  • Advertising sites that pretend to have real content to WWW spiders, but when an actual person with a WWW browser comes along, serve up only advertisements or linkfarms.
  • Subscription-only content sites that present the information that is behind the costwall to the WWW spider, so that it is indexed, but not to people with WWW browsers unless they have subscribed.
  • Sites that don't want to be crawled, and that therefore present blank pages or give error responses to WWW spiders.

As mentioned, there are better ways to do at least the latter two, and so doing things this way is rare in practice. Doing things this way leads to an arms race between the WWW indexing companies and the WWW site providers.

On the gripping hand, this won't combat your "Burp Site crawler" anyway. It isn't initially a WWW spider at all, but a passive data analyzer that works off traffic, generated by humans with WWW browsers, seen through a proxy HTTP server.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .