I would like to know what is the most efficient way of concurrently accessing thousands of files of a similar size in a modern Linux cluster of computers.
I am carrying an indexing operation in each of these files, so the 4 index files, about 5-10x smaller than the data file, are produced next to the file to index.
Right now I am using a hierarchy of directories from
./99/99/99 and I place one file at the end of each directory,
It seems to work better than having thousands of files in the same directory but I would like to know if there is a better way of laying out the files to improve access.