Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I've been looking for a way to tag my files and search/filter them based on those tags.

Here are my (updated) requirements :

  • any file readable by the user can be tagged freely
  • a user can search for files matching one or several tags
  • files can be moved around without losing the previously associated tags
  • the system could be backed up easily
  • no dependencies on any desktop environment
  • if any gui is involved, there must be a cli fallback

I've been hoping for some basic filesystem & coreutils hackery to handle this, but I haven' thought about this hard enough yet.
Meanwhile I'll review beagle and metatracker, which have been mentionned here, and see how they perform.


Ok so beagle has huge gnome dependencies, and tracker is okish, but still has some dependencies I don't like...

Been doing some more research, and the way to go could very well be extended file attributes.
That's a native solution for most recent filesystems, but they aren't very well supported yet (most coreutils destroys them by default, cp for example needs the -a flag to preserve them). Would like to hear some thoughts on using them while I try my hand at some hacks myself, eventhough this might warrant a new question.

share|improve this question
    
Issues with extended file attributes: (i) In my experience, they are a nuisance when you want to backup. (ii) You can't use them when you move between filesystems. Apart from that, they would be the Right Thing. –  Charles Stewart Jan 19 '10 at 9:28
    
PytagsFS superuser.com/a/89140/129520 –  naxa Nov 21 '12 at 10:41

11 Answers 11

up vote 8 down vote accepted
+100

It's not clear what kind of searching you want. If you want it to work anywhere in unix, rather that just your home directory, and you only want to do pathname-based searches, the following scheme is workable, with a little bit of shell hackery, and using the standard locatedb:

  1. Each directory that contains at least one tagged file needs a standard subdirectory, say .path-tags;
  2. Each file in the directory $FILE with link $TAG (which should not contain the char _) has a link $TAG_$FILE -> ../$FILE

I leave the details of the locate-tag script to you; it should be a two- or three-liner, using only the locate command and shell hackery. (If you're interested, I could write one).

Some of the KDE chaps talked about this sort of scheme for metadata, although I don't recall the details.

It should also be possible to do more sophisticated, content-examining tests based on this scheme with a similar script wrapped around find.

Thoughts on updated requirements

  1. any file readable by the user can be tagged freely - Yes, should be no problem
  2. a user can search for files matching one or several tags - Likewise
  3. files can be moved around without losing the previously associated tags - The directories they inhabit can be freely moved about, but if the file is moved from the directory, we are in trouble. If the tags took the form $TAG_$INODE_$FILE and we have an efficient way to find which paths have a given inode, then we can do this, losing tags only if we move out of filesystems. Copying files might make some trouble, and this is clearly more complicated than my original suggestion.
  4. the system could be backed up easily - not essentially difficult.
  5. no dependencies on any desktop environment - none
  6. if any gui is involved, there must be a cli fallback - that's where we live!

Postscript The "reverse-inode-lookup" file described by the link (2) you showed me in your answer to (1) can be used to give some additional infrastructure. We can run a service on the reverse lookup file, which checks that each inode given in the filename of a tag matches the inode of the file (if any) the tag points to. If there is no match, then the required surgery can be performed (does the inode still exists? where is it?), and the reverse lookup file being either mutated or regenerated, and the tag symlinks being updated.

I anticipate one tricky case: what if the tagged file is not where the tags say it should be, the reverse lookup file says it still exists, but the prodigal file is not where the lookup file says it is, the lookup file being out of date? There are a few ways to handle this case, none obviously ideal. Apart from this, this whole task seems to be the kind of thing Perl is well-suited to...

share|improve this answer
    
This is nice, and I've been thinking about using symlinks too. The problem is, a file can't be moved around without losing its tags. Ideally, tags would be path agnostic, and searching for a tag should return the actual file, rather than a dead symlink... PS : I'm all for a shell based solution, but I think the problem domain make it so that It'd be pretty painful to maintain only through shell scripts, I hope someone proves me wrong –  julien Jan 18 '10 at 14:40
    
I've edited my question to (hopefully) make it clearer what kind of solution I'm after. cheers –  julien Jan 18 '10 at 14:54
    
Damn I had never realized that inodes where like persistent guids for files, that's food for thought! –  julien Jan 18 '10 at 17:29
    
inodes are uids, but they are tied to a given fs, so they are not guids. This is not a bad thing, since copying, backups, archiving, &c, mean that files get duplicated and stored within other files, and you want the fs state to give you enough info to disentangle the results. –  Charles Stewart Jan 19 '10 at 11:32
    
This sounds fun, I have quite a fewthings on my plate right now, and can't tackli this at the moment, but I will be experimenting with this too, let's keep in touch. –  julien Jan 21 '10 at 9:27

I've just released an alpha of my new program that attempts to provide this functionality. It currently meets some, but not all, of your requirements. It may be of interest to you anyway. It provides a command-line tool for tagging and a virtual file-system for browsing (where tags are represented by directories).

http://www.tmsu.org/

any file readable by the user can be tagged freely

Yes.

a user can search for files matching one or several tags

Yes. Either via the command-line tool or by browsing the tag directories in the virtual file-system.

files can be moved around without losing the previously associated tags

No. However the application stores fingerprints of the files tagged which are used to help identify moved files. A 'repair' command is provided that will update the paths of moved files. (Obviously this mechanism breaks down if a file is both moved and modified.)

the system could be backed up easily

Yes. It's a simple Sqlite 3 database file.

no dependencies on any desktop environment

Yes. No dependencies and as it can be run as a virtual file-system it is available to peruse as a file-system in any program that supports symbolic links.

if any gui is involved, there must be a cli fallback

No GUI at present.

share|improve this answer
    
Looks very interesting. Do you have any idea how to implement the possibility to move files around without losing the associated tags? –  student Aug 6 '12 at 18:02
    
@student: currently there is a 'repair' command which deals with moved and modified files. (If you both move and modify a file, however, this won't be detected.) –  Paul Ruane Aug 7 '12 at 19:10
    
Perhaps one could write variants of mv, cp and rm which handle your tags as well (call them for example tmv, tcp and trm) then one wouldn't lose tags at least if one uses the commandline to move files around... –  student Aug 9 '12 at 20:57

I think this might meet all your requirements. In any case, it is a cool piece of code:

http://pages.stern.nyu.edu/~marriaga/software/oyepa

The GUI requires Qt, but there is a command-line application for searching and the fact that all tags are actually in the filename makes it trivial to manipulate tags|files from the cli.

share|improve this answer
    
From the page: "Tag information is stored in the filename" - so what do the tagged filenames look like? BTW, the links on that page are very interesting: +1. –  Charles Stewart Jan 19 '10 at 9:28
    
report-for-bill[work stuff,hr,produced by me].odt –  laramichaels Jan 19 '10 at 16:22

You probably don't need to install entire KDE desktop for their tagging library, Nepomuk. You would still have to install KDE base libraries, though...

share|improve this answer
1  
yeah well I was hoping to find an alternative to this, but it doesn't look so... –  julien Dec 11 '09 at 10:54

So you won't find Nepomuk integration in gnome, at the command line, or elsewhere in Linux.

Conversely, with Tracker you won't find kde integration AFAIK. Not sure on CLI.

So unfortunately, the answer appears to be "no".

Even more unfortunately, this doesn't mean there's a good opportunity here for building one either. Linux commandline utilities dont have much in common with the GUI file manager, for example, so architecturally there's no common componentry which could be extended to support the concept.

share|improve this answer

Try Beagle. I find it is pretty good.

It may not meet all the requirements, and I'm not sure what could. For example, do FIFO files support extended attributes? If they don't, Beagle has a fallback database.

share|improve this answer
    
Can beagle handle non-regular files? –  Charles Stewart Jan 18 '10 at 10:46
    
@Charles Stewart - do you mean non-text files? –  pcapademic Jan 18 '10 at 17:13
    
No, I mean device files, symlinks, FIFOs, &c –  Charles Stewart Jan 18 '10 at 18:55
    
That link does not refer to a project about document organisation. –  detly May 15 at 3:58

I too have been working on an easier and simpler file tagging solution, it may help with your feature requirements:

https://github.com/HouzuoGuo/markafile

Some highlights:

  • SQLite as centralized tag database
  • Support complex tag queries
  • Queries are translated into SQL for faster execution

Regarding your feature requirements:

any file readable by the user can be tagged freely

Yes

a user can search for files matching one or several tags

You may use "not", "and", "or" and parenthesis to build complex tag queries.

files can be moved around without losing the previously associated tags

Not yet implemented.

the system could be backed up easily

Single centralized tag database, thus backup and restore is easy.

no dependencies on any desktop environment

It depends on Python 3 and SQLite.

if any gui is involved, there must be a cli fallback

It is designed to be a CLI solution.

share|improve this answer

This recent article on Linux Desktop Search Tools mentions that Tracker supports tagging. Unfortunately it's supposed to be half-broken in the old version they tested. Maybe it's fixed now?

  1. Not system wide.
  2. You can back it up.
  3. It's bundled with Gnome.
share|improve this answer

I made a little program that uses SQLite for this purpose. It solved my need, but maybe it helps you too:

https://github.com/alvatar/dfym

The only issue with this approach is that does not synchronize with moves and deletions, but it solves the problem for relatively static files.

share|improve this answer

Nobody mentioned, but you definitely should look at extended file system attributes. ext4 for example has them. there are tools getfattr and setfattr to deal with them. Of course you will have to write some shell scripts to search for files tagged with sometag. Regarding mentioned questions all the answers are "Yes". You should only take into account that it's depended on file system.

share|improve this answer

I suggest taking a look at a version control system such as Subversion for these kinds of features above and beyond the file system. Some may be a better fit for you than others but generally:

  • Many support tagging (certainly subversion).
  • Many are cross platform; Windows, Mac, Linux, pretty much all Unixes.
  • Many have both GUI front ends and command line clients.
  • Many already have bindings for your favourite programming/scripting language.
  • Many are easily backed up.
  • Many are designed to be very easily shareable in one way or another.
  • Many allow you to control access.
  • You don't have to re-invent the wheel.
    • You learn and use standard commands/tools already used by millions.
  • You can install it today for your favourite OS repo; apt-get install, yum install
  • You also get version management "for free".

A cli example with Subversion: ~/svn/atestrepository: $ svn propset mytag "something" dir1 property 'mytag' set on 'dir1' $ svn propset myothertag "nothing" dir1/file1 property 'myothertag' set on 'dir1/file1' $ svn propset anemptytag "" dir1/file2 property 'anemptytag' set on 'dir1/file2'

$ svn propget -R mytag dir1 - something ~/svn/atestrepository: $ svn propget -R myothertag dir1/file1 - nothing $ svn propget -R anemptytag dir1/file2 - $ svn proplist dir1/file2 Properties on 'dir1/file2': anemptytag svn:keywords

I wouldn't recommend these tools is for large (gigabyte sized) regularly changing binary files but for everything else they are already well proven and scale to very large sizes.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.