Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I run a site which provides Subversion & TRAC for lots of open source projects. This works pretty well with one exception: Many search engines do not care about robots.txt and DoS the webserver with many parallel requests in TRAC, for example the TRAC changesets as tar/zip.

As I run a lot of TRAC repositories under one domain, I use wildcards in robots.txt, which according to Google should be permitted:

User-agent: *
Disallow: /*/changeset
Disallow: /*/browser
Disallow: /*/log

Unfortunately even Google does not care about it, although the webmaster tools confirm that the specific URIs should be ignored. And yes, I told them but they didn't really care. For sure others like Yandex do not care about it either.

So plan B is to lock them out in the Apache configuration, a friend of mine gave me some hints about how to do it:

<Directory /foo/bar>
                SetEnvIf User-Agent Yandex BlockYandex=1
                SetEnvIf User-Agent METASpider BlockMETASpider=1
                SetEnvIf User-Agent BlockMailru=1
                Order allow,deny
                Allow from all
                Deny from env=BlockYandex
                Deny from env=BlockMETASpider
                Deny from env=BlockMailru

Now I try to figure out if I can do something like that with wildcards as well so I don't have to do a <Directory> section for each repository. I found <LocationMatch> in the Apache docs but I am not sure if I can use that as a replacement for <Directory>.

So my question is can I use <LocationMatch> for that and/or does anyone have some better ideas on filtering the bots on server side?

share|improve this question
up vote 1 down vote accepted

(I wanted to answer it inline as recommended by the site but edit of the question lead to an error message)

I checked out more documentation at and figured it out myself:

   # get rid of the bots which are too stupid to respect robots.txt
   <LocationMatch "/[^/]+/(browser|changeset|log)">
      BrowserMatchNoCase googlebot ImBot
      BrowserMatchNoCase Yandex ImBot
      BrowserMatchNoCase bingbot ImBot
      Order allow,deny
      Allow from all
      Deny from env=ImBot


Test with a user-agent-switcher extension for your browser. Due to noob-limitations I cannot post more links to it :) My site is at for those who want to try it live.

share|improve this answer
If you find the solution yourself, it should be posted as an answer like this. It should not be edited into the question. – grawity Sep 1 '11 at 22:26
Ah ok so I did it right :) thanks! – Adrian Gschwend Sep 2 '11 at 9:20

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.