140

I have three computers at home, and would like to have the /home/ folder tree synchronized between the three.
Any files/folders (except those hidden) that are modified/added/removed in one of the three computers is updated in the other two.

Is rsync enough to do this? and how?

3
  • 14
    The problem with most 'syncing' tools, (including rsync if I recall correctly), is that they don't have a notion of user actions. So if you delete a file on computer X, the tool doesn't know whether (a) you actually deleted the file on computer X, or (b) that you added that file on computer Y. Most tools will assume (b), so upon syncing they will add the file again on computer X (instead of deleting it on computer Y). Mar 2, 2011 at 14:11
  • 7
    @Rabarberski many sync tools (including Unison or Syncthing) keep a database with references to the existing data (for this reason). It won't help you when syncing new folders for the first time, but they can very well delete files for you (automatically). The real problem with most sync tools is when files have been changed in two locations before the next sync occurs - in this conflict case you generally have to do a manual merge or maybe replace with the newest version.
    – Chris
    Aug 3, 2015 at 9:06
  • 3
    Syncing when the changes can't be instantaneously copied on the other machines is really tricky and I've never seen any program do it perfectly, so be prepared for a few bugs, conflicts, of reappearing files. I would recommend syncing only some specific folders that you know will be manageable by the syncing program.
    – Hey
    Jun 1, 2017 at 10:56

16 Answers 16

118

Here's a list things that could potentially solve this problem, each balances the trade-offs you have to make differently so you'll have to make your own choices and try things out for yourself:

  • Unison - as mentioned by others, this is run manually, but is very fast, reliable and effective. Requires both machines being synchronised to be on at the same time. It has a nice user interface to allow you to deal with the almost inevitable conflicts, and tracks and propagates deletions correctly. The graphical app/package is called unison-gtk.

  • OwnCloud - Cloud storage run on your own server. You'll need a machine to leave on. Requires a reasonable amount of setup. Runs a full Apache 2 webserver and an SqlLite or MySQL database on the server. Works similar to Dropbox with a desktop client, but the server is under your control. edit : OwnCloud has recently gone through some changes in how the project is run, and now has a new fully open source (ie no closed source 'enterprise' edition) under the guise of NextCloud, (see this youtube interview with the original OwnCloud developer for more details).

  • SparkleShare - uses git to keep files in sync. According to the homepage: good for many smaller files, not good for lots of large files such as music or photo collection.

  • Seafile - Provides a server component you can install on a local machine. Seafile uses a data model similar to git for tracking changes. Provides sync clients for desktops, tablets and smartphones. A blog post describing setup can be found at http://openswitch.org/blog/2013/07/18/installing-and-configuring-seafile-on-ubuntu-12-dot-04/

  • Osync - "... bidirectional file synchronization tool written in bash and based on rsync. It works on local and / or remote directories via ssh tunnels. It's mainly targeted to be launched as cron task" (text from the website)

  • PowerFolder - java based GPL v2 project. Main website pushes commercial offerings so it's not clear how to use the provided .jar file.

  • Rsync - fast and effective and been around for decades, however it doesn't keep a history so you have to choose a direction to decide whether a file is new or deleted. Graphical tools are available such as gwRsync.

  • Lsyncd - monitors folders/files to trigger rsync replication

  • dvcs-autosync - written in python, uses git to store and share changes between machines, and XMPP to communicate changes.

  • git-annex - command line tool for shunting files around, based on git. There's an illustrative walkthrough here: http://git-annex.branchable.com/walkthrough/

  • Tonido - freeware. Provides a desktop app that will share files to other devices. Also provide commercial cloud offerings, and the TonidoPlug plug computer.

  • BitTorrent Sync (freeware) - peer-to-peer file sync based on BitTorrent. I don't know much about this as I won't be using it due to it not being open source and not trusting it to keep my data within my LAN, feel free to edit this answer with better information / real experiences.

  • SyncThing - Developed as an open source alternative to BitTorrent Sync. It currently lacks some of the advanced features of BitTorrent Sync, such as untrusted peers. It is under active development.

  • Commercial hosted services such as dropbox, ubuntu one, google drive, apple iCloud are all quick cheap and convenient, however they all require trusting a company with all your data, and need a reasonably fast internet connection.

  • Git / subversion - Use a source control system directly. Completely manual and can be a little complex but popular approach with some users familiar with these systems from using them as programming tools.

  • CloudFS - syncronise a whole filesystem, cluster technology based

  • NFS mount - basically your home lives on one machine and you access it over the network, no good for laptops you take with you. More info: http://www.linuxjournal.com/article/4880


Factors to consider in making your decision:

  • Central server - some solutions require a machine to be on all the time (or at least when you need to synchronise) for other machines to synchronise with. This could be one of your existing machines, or a separate machine such as a NAS. Watch out for increased power bills.

  • Automatic / Manual / Scheduled - The best way to avoid having to resolve conflicts where something is changed on more than machine is to have a program on every machine that watches for changes and synchronises immediately, this way you reduce the opportunity to end up with multiple versions. With manual processes you always have to remember to run the synchronisation.

  • Remote access - do you want to synchronise away from your LAN (aka home), think about the security implications of this.

  • Security - does your data leave your network encrypted or not, how secure is the transfer between machines. What if someone captures your data on the move and later the encryption is found to have flaws? Who controls the server that keeps your data, is the data enccrypted, can you trust any third-parties? Do you have to poke holes in your router to get remote access. How long do 'deleted' files and related meta-data stick around for on the synchronised devices and on the central server. Are you synchronising between encrypted and unencrypted storage?

  • Moving large folders - the solutions I've tried all have an issue that when you move / rename a file or folder the sync doesn't understand this and uploads it all over again as new and then deletes the old copy.

  • Disk capacity

  • Backups - synchronisation is not backup. Delete an important file by mistake and many of the above will merrily delete all your other copies. I recommend reading Mat Honan's piece on being hacked for a good account of what can happen if you put all your digital eggs in one digital basket, so to speak.


I recommend not syncing the entire home folder, but instead picking specific folders to sync such as Documents/, Pictures/ etc. This will avoid the pain of being forced to deal with the speed / performance / disk space issues of automatically synchronising everything. It also avoids having to maintain exclusion lists.

As I continue to try and find something that works for me personally I'll try and keep this answer up to date with useful info. I've aggregated the information from all the other answers into one complete answer.

References:

  • LinuxFormat - February 2014 LXF180 p31, "Hosted Storage Roundup"

hacking/all/

2
  • I have a server w/out root access (which is needed to install unison, for example). Is there a script I can have on my client machine that doesn't need to be installed on my server? Aug 2, 2015 at 19:17
  • I've seen a software named freefilesync (sourceforge.net/projects/freefilesync) mentioned a few times here and there. Might be worth mentionning in your list ?
    – YoungFrog
    Aug 3, 2015 at 6:54
27

Unison might be a good candidate:

Unison is a file-synchronization tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other.

It already does 2-way syncs. See update bellow.

I've learnt that there are very few things rsync cannot do, and it can probably provide an equal or better solution, but you'll have to wait for an rsync expert to turn up for that solution.

Update: Yes, Unison can sync more than 2 machines. From their user manual:

Using Unison to Synchronize More Than Two Machines

Unison is designed for synchronizing pairs of replicas. However, it is possible to use it to keep larger groups of machines in sync by performing multiple pairwise synchronizations.

If you need to do this, the most reliable way to set things up is to organize the machines into a “star topology,” with one machine designated as the “hub” and the rest as “spokes,” and with each spoke machine synchronizing only with the hub. The big advantage of the star topology is that it eliminates the possibility of confusing “spurious conflicts” arising from the fact that a separate archive is maintained by Unison for every pair of hosts that it synchronizes.

3
  • I have some bad experiences with unison, for some reason from time to time instead of adding the new files from other computers, it deleted them while other times it worked fine.
    – jhcaiced
    Nov 29, 2011 at 20:48
  • 1
    I suggest editing the a/b/c example to follow the recommendation in the manual not to create loops.
    – Tim Abell
    Jan 18, 2014 at 16:47
  • Regarding your reference to rsync. No it can not reliably do two way sync. Look at the following answer to see why unison can handle a case that rsync can't: stackoverflow.com/questions/4504357/…
    – ndemou
    Oct 7, 2015 at 11:02
20

How about to put your files in a version control system like SubVersion or git?

I don't version my entire home dir, but only a handful subdirs with some important stuff. But I am thinking about converting to git since it seem to be better for this type of job. (a quick search is going to send you in the right direction).

Good luck


Update: A nice side effect with git is that it is easy to have computer specific files, since you only need to have a branch for that computer (and in git you get that by default). So you don't need to have exactly the same files in all the computer, they only need to be similar, and have a common "main" branch with stuff that you want on all the different computers.

6
  • Yikes. The idea of using version control for file sync really makes my skin crawl.
    – JohnD
    Nov 12, 2011 at 1:59
  • Yes the idea is strange but really useful. Think about a dir like ~/bin/ for private helper scripts, that kind of dir is great in a version control system. While a dir like .kde will probably behave bad in the version control system and would be put in a ignore list for file not handled.
    – Johan
    Nov 12, 2011 at 8:04
  • This is a great way to keep your files synched, you have to get used to the idea of using "git add" and "git rm" to add/remove files but works very well.
    – jhcaiced
    Nov 29, 2011 at 20:51
  • I like this idea because I'm in control what part of files within garbage that I have in my home will be shared. The only question is how to manage commits. i.e. some files should go into initial commit and then reflect changes to show that they based on system /etc/skel.
    – ony
    Sep 28, 2013 at 6:46
  • Is maintaining a couple of separate branches with your files really convenient, when you are trying to sync directories? Sure, it works, but I wouldn't recommend (g)it for this task. You can generally exclude files from most sync tools (either by name, folder, etc) and for what you are describing (using it just for some important sub-directories), it's not really useful in regards to the original question.
    – Chris
    Aug 3, 2015 at 9:16
6

Your main problem when trying to do this is deciding how to merge changes, propagate deletes, and resolve conflicts. This is hard to do in a completely automated way, especially if you've got a 3 computer setup used by multiple people.

If you separate users things get much simpler. Because one user can't be in two places at once (and therefore generate conflicts) you could then setup an rsync job to run on login to "get changes" and logout to "push changes" ... to one of your computers which would be the master ... so at this level of granularity you'd be syncing /home/myuser each time rather than the entire /home/. An added refinement (in case of people not logging out) would be run the push script after a short period of inactivity.

There are lots of other potential solutions, but none that magically solve this problem as far as I'm aware. The first step is probably to think about how you use the machines and come up with a syncing policy to suit your user's behaviour.

6

I think you can achieve what you want better by NFS mounting a common home folder. Check out this article http://www.linuxjournal.com/article/4880

4

This won't entirely provide a solution, but it will give you a start:

Set up a cron job every so often to rsync the files. I use a command like the following:

rsync -alhz --stats --progress --exclude-from '/etc/rsync_backup_excludes.conf' / -e ssh [email protected]:/path/to/home/

This will use rsync to only copy the needed changes, not re-copy everything every time the command is ran.

--stats and --progress are optional and should probably be excluded from your scripts

the contents of my rsync_backup_excludes.conf file are:

dev/*
proc/*
lost+found/*
mnt/*
sys/*
2
  • Anacron will sync when the machine is on, avoiding missing slots due to downtime.
    – Tim Abell
    Jan 18, 2014 at 18:06
  • This copies your entire local root into the remote home folder. That's not what the OP asked for, nor can I imagine why anyone would want that...
    – Cerin
    Oct 31, 2015 at 3:02
4

Check out lsyncd

Lsyncd watches a local directory trees event monitor interface (inotify or fsevents). It aggregates and combines events for a few seconds and then spawns one (or more) process(es) to synchronize the changes. By default this is rsync. Lsyncd is thus a light-weight live mirror solution that is comparatively easy to install not requiring new filesystems or blockdevices and does not hamper local filesystem performance.

Rsync+ssh is an advanced action configuration that uses a SSH to act file and directory moves directly on the target instead of retransmitting the move destination over the wire.

Fine-grained customizaton can be achieved through the config file. Custom action configs can even be written from scratch in cascading layers ranging from shell scripts to code written in the Lua language.layers ranging from shell scripts to code written in the Lua language.

2

Take a look at dropbox http://www.getdropbox.com/ Cross platform Win / Linux / Mac

2
  • 4
    Thanks, but no. Tried it already. It's not what I'm looking for.
    – lamcro
    Aug 28, 2009 at 18:46
  • Dropbox is great and I use their free 2GB for storing links, photos, and other non-mission critical data between home, work and my laptops. It isn't what you need in this situation, but in my opinion, Dropbox definitely has its place.
    – Matt Cofer
    Aug 28, 2009 at 18:59
2

Mount /home from one computer over to the other two. Automount works pretty well for this.

1

Ubuntu One might be what you're looking for. Unfortunately it's still in beta phase, and I have no experience with it, so I'm not sure if it'd work for you.

1
  • I'd say it's still in heavy development at the moment - there are updates every few days as the Ubuntu team iron out the kinks. It works, but I'd not recommend it over Dropbox.
    – user4358
    Sep 1, 2009 at 20:59
1

If you also want to sync configs - tools mentioned above are totally helpless: configs are often changed, and many logfiles will make conflicts so they can't be merged.
My deceision is simple, and stupid :) I've got a Master workstation, and all files from "~" are just copied to the second "Slave" one. The moment i realize I need to modify something - i do it on Master, and Slave catches these changes on sync.

Also, some bash scripts should run different on these machines, so I edited my /etc/bash.bashrc:

export OOHOST=Master

Now, scripts know which host they're serving ;)

1

Try dvcs-autosync.

This is based on git, uses XMPP to notify conected clients of file changes and reacts on file change events via inode changes. So it always gets informed right after the file change, in case of conflicts it relies on the proven methods of git.

I've been using it for a while now as a Dropbox replacement via SSH public keys and it really does the job.

0

PowerFolder - Sync Files, Sync Folders, Remote Storage, Backup and Private File Sharing. Sync home and office PC, share holiday pictures or work together on documents. PowerFolder's secure peer-to-peer technology works over the Internet or in LAN.

0

The workarounds I would consider:

  • Dropbox.
    Main drawbacks: it's not free for any reasonable amount of data. Otherwise it works flawlessly IMHO

  • A NAS server. Put your files on a NAS server, they became reasonably affordable lately (from 150 $ or so) and a good one doesn't consume much energy. No synching issues ever.
    Main drawback: slower than a local copy

1
  • With the NAS, you're still not addressing how the synchronization happens. Having a NAS and two computers is, in principle, no different from having three computers as OP is asking about. If you're suggesting that everything that needs to be synchronized is just stored on the NAS, then that's not generally possible - consider synchronization of config files for instance - they need to be on each machine.
    – kba
    Jun 22, 2012 at 16:43
0

I use SyncTwoFolders for Mac and Syncback for PC - the first one for my three macs at home (inclusive laptop), and the second for my PC at work. Both program have various back-up and sync options making this a breeze once pre-sets have been set up. Easy-peasy! I'm sure there are other versions of similar software available for Linux and Ubuntu.

0

You could use a sync script like osync which can propagate deleted files and update only modified ones. http://www.netpower.fr/osync osync is rsync based but can handle a lot of sync scenarios between local folders or remote ones over ssh.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .