Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

Let's say I have 100,000 files on a computer. And they are not too big, but not too small either. Let's say around 50MB each. But they can be all different sizes from very small to 5-6GB.

Note these numbers are arbitrary, I don't have those files. Everything is theoretical.

I was wondering what is the best folder setup for storing those files? And does it differ for different operating systems? Does it depend on the filesystem? Can an imaginary operating system exist that doesn't store information on the HDD as files and folders? Or is this files and folders structure dependent on the hard drive's filesystem format? Is storing them in one folder a good idea? Or is it better for storing them in a height-balanced tree? Or in weight-balanced tree? What are the drawbacks for having many folders? How much space does it take to have a folder (I know it's a little)? What are the drawbacks of having a very deep directory structure? Are there any existing solutions for managing many files? Does the best solution depend on the number of files and the file size directly? Is there an optimal solution that does not involve file size and file number?

I know that these are a whole lot of questions.

Also note that the files are going to be browsed by software and not by an user directly.

Relevant questions:
How many files can a windows folder contain?
How to organize a directory with many different file types?

Why am I asking all those questions about files, folders and filesystems?

Well, I am planning to make a program that runs on any OS and is something like file-explorer. You can have files and folders, but on the OS they will be stored differently. For example I have 50 files in one virtual folder, and the program manages to arranges them on the hard drive in sub-folders.

share|improve this question
    
You're asking for a recommendation. Your question will likely be close unless you edit it considerably. Please see help center. We don't do recommendations, we solve actual technical problems. –  user 99572 is fine Aug 4 '13 at 12:57
    
@user99572isfine No, I am not asking for a recomendation, I ask for optimal solution. Read my entire question and then speak. –  Bosak Aug 4 '13 at 12:58
    
You're asking about low-level file system issues and managed to not even mention what OS you're using? That information would probably be quite helpful... –  Daniel Beck Aug 4 '13 at 12:58
    
@Bosak Don't get rude –  nixda Aug 4 '13 at 12:59
1  
Please narrow down your question. Right now, it's not a lot better than "Tell me everything about file systems". It might help if you specifically explained what kind of data you want to manage or make available in in what way. Right now, a feasible approach would be to just rename all files to their SHA checksum and keep separate records on logical file paths and file checkums -- there'd be no need to enumerate files, for example. –  Daniel Beck Aug 4 '13 at 13:07
show 8 more comments

closed as too broad by afrazier, Dave M, Journeyman Geek, teylyn, Mokubai Aug 5 '13 at 5:55

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

1 Answer

The file system usually gives a tree decomposition of an array of files into pieces. Accordingly, there come to the aid of taxonomy. Criterial or other methods of decomposition of the whole into parts.

The operations that are performed on file content - instantaneous search - he's a full-text search.

Giving file object tags to overcome the limitations of the tree structure.

Create files of a certain type of text content.

For example the data stored in the database are transformed into: xml, pdf, ms-docx etc.;

State the purpose. Specify the type of data contained in the files and you can continue.)


Files can be divided into groups.

The first group consists of text content and layout

The second is the video content and audio files - Media Library.

Third - Special files - such as music library, or a file with a special focus content - such as literature with chemical or mathematical formulas.

Fourth - search queries and data mining resources in the Internet.

Accordingly, any problem related groups.

Common tasks for all: Generation of the desired file format from the database. (Xml, pdf, ms-docx, ps ...)

Indexing. Full-text search.

Tagging. Automatic Tagging based on content analysis.

First group:

Splitting a text file into the simplest components (text, images, tables, links, diagrams) and markup. Translated to obtain atomic information in the database.

Automatic Tagging based text analysis file.

Second group:

Tagging library. Comparison of video files. Voice Recognition. Adding content and associated hyperlinks.

Third group:

Creating a search engine for specific content types. Translated special content in the data on which you can search.

Fourth group:

Collecting information through search engines.

share|improve this answer
    
"Giving file object tags to overcome the limitations of the tree structure" - That was exactly on what I was planing to do. To give files tags and tags to be like 'folders' so you can have one file on many places, but it will be only a reference to the actual file. And the file-explorer will automatically manage the internal storage. –  Bosak Aug 4 '13 at 13:36
    
About the type of data, well anything can go. From text files, images, music, huge files and small ones too. –  Bosak Aug 4 '13 at 13:37
    
@Bosak updated ... see too garshol.priv.no/blog/231.html and en.wikipedia.org/wiki/Triplestore –  STTR Aug 4 '13 at 17:20
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.