Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

OK, currently we are restricting the directory /Assets/ from being indexed in our robots.txt. There is one directory under the /Assets/ directory that I want indexed (/Assets/product_labels/). Is there any way to allow this one directory, other than moving it to another directory?

share|improve this question

2 Answers 2

up vote 1 down vote accepted

You could put explicit Disallow for all files and folders that are 'siblings' to the folder you want to keep open.

Disallow: /Assets/Dir1
Disallow: /Assets/Dir2
Disallow: /Assets/DirN
Disallow: /Assets/File1
Disallow: /Assets/File1
Disallow: /Assets/FileN

Just leave the /Assets/product_labels out -- to allow it.

Generation of this list can be automated, but remember this has to be done for every new directory or file you keep under Assets that you want to disallow.

Of course, if you are good to move the product_labels out of Assets it will be cleaner.

share|improve this answer
    
Thanks for the feedback. –  xxl3ww Jul 30 '09 at 13:52

" This is currently a bit awkward, as there is no "Allow" field. The easy way is to put all files to be disallowed into a separate directory, say "stuff", and leave the one file in the level above this directory: " --> Quoted from: www.robotstxt.org

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.