Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I run a site which provides Subversion & TRAC for lots of open source projects. This works pretty well with one exception: Many search engines do not care about robots.txt and DoS the webserver with many parallel requests in TRAC, for example the TRAC changesets as tar/zip.

As I run a lot of TRAC repositories under one domain, I use wildcards in robots.txt, which according to Google should be permitted:

User-agent: *
Disallow: /*/changeset
Disallow: /*/browser
Disallow: /*/log

Unfortunately even Google does not care about it, although the webmaster tools confirm that the specific URIs should be ignored. And yes, I told them but they didn't really care. For sure others like Yandex do not care about it either.

So plan B is to lock them out in the Apache configuration, a friend of mine gave me some hints about how to do it:

<Directory /foo/bar>
                SetEnvIf User-Agent Yandex BlockYandex=1
                SetEnvIf User-Agent METASpider BlockMETASpider=1
                SetEnvIf User-Agent Mail.ru BlockMailru=1
                Order allow,deny
                Allow from all
                Deny from env=BlockYandex
                Deny from env=BlockMETASpider
                Deny from env=BlockMailru
</Directory>

Now I try to figure out if I can do something like that with wildcards as well so I don't have to do a <Directory> section for each repository. I found <LocationMatch> in the Apache docs but I am not sure if I can use that as a replacement for <Directory>.

So my question is can I use <LocationMatch> for that and/or does anyone have some better ideas on filtering the bots on server side?

share|improve this question
up vote 1 down vote accepted

(I wanted to answer it inline as recommended by the site but edit of the question lead to an error message)

I checked out more documentation at Apache.org and figured it out myself:

   # get rid of the bots which are too stupid to respect robots.txt
   <LocationMatch "/[^/]+/(browser|changeset|log)">
      BrowserMatchNoCase googlebot ImBot
      BrowserMatchNoCase Yandex ImBot
      BrowserMatchNoCase bingbot ImBot
      Order allow,deny
      Allow from all
      Deny from env=ImBot
   </LocationMatch>

References:

Test with a user-agent-switcher extension for your browser. Due to noob-limitations I cannot post more links to it :) My site is at svn.netlabs.org for those who want to try it live.

share|improve this answer
    
If you find the solution yourself, it should be posted as an answer like this. It should not be edited into the question. – grawity Sep 1 '11 at 22:26
    
Ah ok so I did it right :) thanks! – Adrian Gschwend Sep 2 '11 at 9:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .