Unrestricting a sub directory in robots.txt

Question

OK, currently we are restricting the directory /Assets/ from being indexed in our robots.txt. There is one directory under the /Assets/ directory that I want indexed (/Assets/product_labels/). Is there any way to allow this one directory, other than moving it to another directory?

Answer 1

You could put explicit Disallow for all files and folders that are 'siblings' to the folder you want to keep open.

Disallow: /Assets/Dir1
Disallow: /Assets/Dir2
Disallow: /Assets/DirN
Disallow: /Assets/File1
Disallow: /Assets/File1
Disallow: /Assets/FileN

Just leave the /Assets/product_labels out -- to allow it.

Generation of this list can be automated, but remember this has to be done for every new directory or file you keep under Assets that you want to disallow.

Of course, if you are good to move the product_labels out of Assets it will be cleaner.

Answer 2

" This is currently a bit awkward, as there is no "Allow" field. The easy way is to put all files to be disallowed into a separate directory, say "stuff", and leave the one file in the level above this directory: " --> Quoted from: www.robotstxt.org