OK, currently we are restricting the directory /Assets/ from being indexed in our robots.txt. There is one directory under the /Assets/ directory that I want indexed (/Assets/product_labels/). Is there any way to allow this one directory, other than moving it to another directory?
You could put explicit
Just leave the
Generation of this list can be automated, but remember this has to be done for every new directory or file you keep under
Of course, if you are good to move the
" This is currently a bit awkward, as there is no "Allow" field. The easy way is to put all files to be disallowed into a separate directory, say "stuff", and leave the one file in the level above this directory: " --> Quoted from: www.robotstxt.org