How to annotate files - when long filenames are not enough

Question

I work with many files doing general data analysis.

Things I want to know about my files include:

• what data is contained in the file (in long and very long descriptive, english text)?
• is the file downloaded from somewhere (where? when?) or generated by a program (which one?)
• why I made this file, verbal description what I want to do with it, where it belongs in my data analysis workflow (additional english text description, can get very long as well)

For this, long filenames are simply not the solution! Even long filenames are too short for the full descriptions, and when actually working with the files (perl, awk, R) the long filenames get in the way.

What I do right now is make a readme in each dir with the filename, tab-separator, and the long description. However this solution is very cumbersome as you can imagine because the descriptions are completely separated from the filesystem and everything, the readme has to be maintained and updated separatedly etc.

Is there any tool one can use for really verbose, systematic descriptions of filenames? Maybe even integrated into the filesystem?

Operating system used: Windows 7 and Cygwin, various flavours of linux/unix through SSH and importing X

Answer 1

As you need a platform independent solution using the file system meta data systems file it is provided by NTFS can not be used.

An alternative would be a database system that manages files and meta data - totally independent of the file-system.

A simple file-system based solution would be to add a second file containing the meta data. E.g. an XML based file with the same filename appended with xml. All current file-systems support multiple file extensions. The additional XML file would have the advantage that a user can read it in an text editor where as tools could also interprete it and extract relevant data like URL from which you downloaded the file.

Answer 2

This answer is in response to your comment about version control systems.

Yes, version control systems can be used for this purpose.
Here's how you could describe a file using git:

$ git init # skip this step if you're already using git
Initialized empty Git repository ...
$ touch abstract.md # create a blank markdown document
$ git add abstract.md # stage this change so that it can be committed to the repository
$ git commit

This will open your text editor defined by $EDITOR, which looks like this:

Create file abstract.md

This file holds my abstracts, and this description         
restates the short message because I can't think of anything better to write.

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
# On branch master
#
# Initial commit
#
# Changes to be committed:
#   new file:   abstract.md
#

When you close the file, git will commit your changes to the repository.

Then to view your commits for that file, do
git log --follow abstract.md.

commit 57bd5244574cd77ae62e17651578d88ee8bca919
Author: Robert 
Date:   Thu May 18 2017

    Create file abstract.md

    This file holds my abstracts, and this description
    restates the short message because I can't think of anything better to write.

I'm not sure if git log --follow follows across branches though.