My Personal Information Management problem

I have gathered terabytes of data but it has became a habit to store files and folders to the same folder, that folder could be kind of seen as a Inbox where most files (non-installations) enter my system. This way I end up with a big collections of files that are hard to organize properly, I mostly end up making folders that match their file type but then I still have several gigabytes of data per folder which doesn't make it efficient such that I can productively use the folder.

I'd rather do a few clicks than having to search through the files, whether that's by some software product or by looking through the folder. Often the file names themselves are not proper so it would be easier to recognize them if there were few in a folder, rather than thousands of them.

Scaling in the structure of directory trees in a computer cluster summarizes this problem as following:

The processes of storing and retrieving information are rapidly gaining importance in science as well as society as a whole [1, 2, 3, 4]. A considerable effort is being undertaken, firstly to characterize and describe how publicly available information, for example in the world wide web, is actually organized, and secondly, to design efficient methods to access this information.

[1] R. M. Shiffrin and K. B¨orner, Proc. Natl. Acad. Sci. USA 101, 5183 (2004).
[2] S. Lawrence, C.L. Giles, Nature 400, 107–109 (1999).
[3] R.F.I. Cancho and R.V. Sol, Proc. R. Soc. London, Ser. B 268, 2261 (2001).
[4] M. Sigman and G. A. Cecchi, Proc. Natl. Acad. Sci. USA 99, 1742 (2002).

It goes further on explaining how the data is usually organized by taking general looks at it, but by looking at the abstract and conclusion it doesn't come with a conclusion or approach which results in a productive organization of a directory hierarchy. So, in essence, this is a problem for which I haven't found a solution yet; and I would love to see a scientific solution to this problem.

Looking for an empirical solution to this problem

Upon searching further, I don't seem to find anything useful or free papers that approach this problem so it might be that I'm looking in the wrong place. I've also noted that there are different ways to term this problem, which leads out to different results of papers. Perhaps a paper is out there, but I'm not just using the same terms as that paper uses? They often use more scientific terms.

I've once heard a story about an advocate with a laptop which has simply outperformed an advocate with had tons of papers, which shows how proper organization leads to productivity; but that story didn't share details on how the advocate used the laptop or how he had organized his data. But in any case, it was way more useful than how most of us organize our data these days...

Advice me how I should organize my data, I'm not looking for suggestions here. I would love to see statistics or scientific measurement approaches that help me confirm that it does help me reach my goal.

Search does not solve this problem

Search is not a solution to this because it disallows access to relevant files, which is often required as you are working in the context of a file (which often requires other files). This is why I mentioned before that bad file names are a mayor problem. And no, putting all directory names in the file name is not the right approach to this. Full text data search engine searches only work for textual files, and the directory names are often not within the file. That makes Search unreliable and uncertain when searching...

Also note that I am looking for structural organization/accessing, because semantic approaches (like "this is how I divide it and works for me and my friends" or "search works for me") are not applicable to everyone, while structural approaches are applicable to the majority of users that navigate their storage.

  • 1
    Found this paper and am reading through it, I might self answer in the future but am still wondering if there is other information on this subject available. At first sight, it does seem to contain some useful information that partially solves the problem... Apr 8, 2012 at 12:06
  • "What was scientifically shown to support productivity when structural " --> sounds more skeptics material.
    – Sathyajith Bhat
    Apr 8, 2012 at 12:57
  • 1
    @Sathya: False, because that side is for researching the evidence behind claims you encounter as per the FAQ. Apparently, you are not really respecting your community. I'd wish to know why this computer data question is off-topic, because otherwise its just you that wanted to see it closed for no clear reason. Apr 8, 2012 at 13:01
  • Furthermore, I want this answer to be on Super User, "empirical or scientific must go to Skeptics.SE" is not the right way to handle these type of questions and simply mean that you are pushing off your highest quality questions to another community. If you decide to go through with this, it sounds like the right time to leave Super User because it has became a mess of close-minded people that do not care about community and their questions. Quality > Quantity... Apr 8, 2012 at 13:04
  • 3
    this isn't about computer hardware or software, this about asking which is a "scientifically shown to support productivity" which clearly is off-topic.
    – Sathyajith Bhat
    Apr 8, 2012 at 13:05

1 Answer 1


Here is a measurement I have done:

  • Clicking through a very deep folder structure, and eyeballing till you find the right file.
    34 secs

  • Pressing the Windows key or Command+Space key, and keying in the keywords for file name.
    9 secs (SSD)

Most modern data search engines searches inside files too, so file names are not really a major concern. [cf OP: Not all files are textual or contain the directory names, they can't be reasonably put in the file name or support context]

As a side note:

Data organization methodology serves two end objectives [Missing proof].

  1. Efficient retrieval.

  2. User OCD.

If it's the latter you have to contend with, then you'll have a problem. If it's the former, a good search engine will allow you to reach the objective just as well or even better [Missing proof].

Also, as more and more information gets downloaded into our lives daily, there will be a point where organizing becomes too cumbersome. [Missing proof]

  • The second method is not productive as it disallows to access relevant files, which is often required as you are working in the context of a file (which often requires other files). This is not scientific as this is just a single measurement (personal observation) but does not helps with navigation and doesn't help with bad file names either. And no, putting all directory names in the file name is not the right approach. Data search engine searches only work for textual files, and the directory names are often not within the file. That makes it unreliable and uncertain when searching... -1 Apr 8, 2012 at 11:55
  • Your two added paragraphs don't answer the question and are just rant. As I outlined, Search does not help if you are disorganized. Please provide empirical proof for your last paragraph (and actually, your whole answer), because it's nowhere near scientific, but rather just a made up thought until you back it up... Apr 8, 2012 at 12:04

Not the answer you're looking for? Browse other questions tagged or ask your own question.