robots.txt. That's a coöperative mechanism for WWW spiders to recognize sites that don't want to be crawled. Actually rejecting the crawler is a slightly different thing, that from a wide perspective few WWW sites do (since
It is done by the content HTTP server for the site "knowing" the IP addresses,
User-Agent: headers, or other information associated with a known WWW spider, and behaving differently when it recognizes a spider requesting a page/file/image.
There are several major variations on this theme:
- Advertising sites that pretend to have real content to WWW spiders, but when an actual person with a WWW browser comes along, serve up only advertisements or linkfarms.
- Subscription-only content sites that present the information that is behind the costwall to the WWW spider, so that it is indexed, but not to people with WWW browsers unless they have subscribed.
- Sites that don't want to be crawled, and that therefore present blank pages or give error responses to WWW spiders.
As mentioned, there are better ways to do at least the latter two, and so doing things this way is rare in practice. Doing things this way leads to an arms race between the WWW indexing companies and the WWW site providers.
On the gripping hand, this won't combat your "Burp Site crawler" anyway. It isn't initially a WWW spider at all, but a passive data analyzer that works off traffic, generated by humans with WWW browsers, seen through a proxy HTTP server.