I am using rsync to keep a large shared document folder in sync with my laptop. More specifically, I am rsyncing FROM my laptop TO the shared repo. What I would like to do is rsyncing only those files that have been created/modified by my user. Being a shared repository other people might have created other directories or files.

I have browsed through the various rsync options but I haven't found what I am looking for. I can --exclude or --delete-excluded but only giving patterns to match the filename, not any other metadata (or can I?).

An alternative, but somewhat cumbersome solution could be launch a remote script like find -not -perm $USER > excluded_files and rsync excluding those, but I'd rather not do that.

Any ideas?

update: to clarify, the data on my laptop will be only part of the data on the server. Here is an example

  • my laptop contains dir_1 and dir_2
  • I rsync them to the server; now the server contains two dirs
  • another user logs onto the server and creates a new directory dir_3
  • I mess up with my files, now will rsync again but need to use --delete
  • I want to --delete ONLY the files that I created on the first place, i.e. those on dir_1 and dir_2, not dir_3
  • Other people might have created files inside your folders? Why don't you create a folder called lorenzog that nobody else writes to? – user39559 Sep 14 '10 at 15:05
  • @lorenzog: Your workflow seems strange: as I understand it, you have files on your laptop that are not owned by you, and you want to copy only the files that are owned by you to the shared repository. Is this right? Do you ever copy files in the other direction? – Gilles 'SO- stop being evil' Sep 14 '10 at 20:53
  • I should explain the situation better. See edit – lorenzog Sep 15 '10 at 12:15
  • And the other users will not create files inside dir_1 and dir_2? – user39559 Sep 15 '10 at 13:57
  • @user39559, yes, they might do, and that's why I was hoping rsync could locate those files and prevent me from deleting them. – lorenzog Sep 15 '10 at 14:02

I assume that keeping a copy of all the files on your laptop is not an option. Otherwise, it would be a lot easier.

The right tool here is a version control tool. But I realize that unless all users have a minimum amount of technical sophistication, getting them to use version control tools is not a realistic suggestion. And even then you'd have to carefully pick what you check out on your laptop.

You might still use version control. On the shared machine, check in all of your files, and leave other people's files alone. Keep a separate checkout on your laptop. Commit and check out whenever you switch machines. The more I think about it, the more I like this option: it doesn't require any coding or even any fragile scripting.

You do two-way synchronization. Rsync is not good at this: if you forget to run it, or accidentally run it in the wrong direction, you're likely to lose work with no warning. Unison is a good tool for two-way synchronization: it always synchronizes bidirectionally, and it complains if a file has changed on both sides. It's open source, and well integrated into both unix and Windows platforms.

Unfortunately, unison doesn't have an option to ignore files by owner, any more than rsync does. You can generate an ignore list with find -user, but it gets messy, and there's a race condition if someone else adds or delete files between the run of find and the run of unison or rsync.

You could patch rsync or unison to add the option to ignore files based on their metadata. I just had a quick look at the source, and in both cases the exclusion code is strongly tied to strings, not to directory entries. It looks doable, but it's not a trivial patch.

You could create a view of the filesystem that contains only your files. I don't have a complete, seamless solution, but here are a couple of ideas to get started.

  • If the shared machine supports hard links (all unices do, and Windows does since NT4 as long as you use NTFS), it's easy enough to create a copy of the repository containing only hard links of your files:

    cd /shared/repository
    find . -user "$USER" -exec sh -c 'mkdir -p "/my/view/${0%/*}" &&
                                  ln "$0" "/my/view/$0"' {} \;

    or with zsh:

    autoload zmv
    zmv -L -Q '/shared/repository/(**/)(*)(u$UID)' "/my/view/$1$2"

    You'd have to be careful not to break hard links when editing or synchronizing. I don't know how rsync and unison would cope. After the initial creation, a file in the shared repository that is owned by you and has a link count of 1 is to be deleted, and a file in your view with a link count of 1 is to be linked in the shared repository. This is all quite clumsy, and I would welcome suggestions for improvement.

  • If the shared machine supports FUSE, you could use it to create a live view of the shared repository containing only your files. I don't know of any existing FUSE filesystem that can do it, but bindfs comes close (I haven't looked at the code to evaluate the coding difficulty).

  • You could use LD_PRELOAD to override readdir (or a Windows equivalent) when running rsync or unison, so that it would only return files owned by you.

  • Brilliant. Thanks for the link, did not know about its existance. Version control was no longer an option once we started uploading large objects (2gb movies, various Microsoft Office documents with large images in them, and the like). – lorenzog Sep 15 '10 at 20:09
  • Unison will work if you can teach it to ignore files belonging to other people. Unless lorenzog wants to mirror other people's files on his laptop and forgot to specify that in the question. If this is the case that lorenzog wants to mirror other people's files on his laptop as well, then it requires nothing from the other users. They don't even need to know that you are updating it with a sync tool. – user39559 Sep 16 '10 at 12:10
  • @user39559: Doh, you're right. I stand by my recommendation to use unison for two-way synchronization in general, but it doesn't actually solve lorenzog's problem. Unison only supports ignoring files by name. Let me think about it some more... – Gilles 'SO- stop being evil' Sep 16 '10 at 17:43
  • Nice solution! But if any change whatsoever is made to the inode when you create hard links, Unison will probably scan everything again and take hours instead of seconds, putting heavy charge on the server. Honestly, this is too an ackward setting where someone wants to share folders so intimately with other people but mirror only his own files. – user39559 Sep 17 '10 at 9:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.