3

I need to setup a synchronization between my Computers with a NAS in the local network. The setup is somewhat complicated, though, so I am assuming there is not an off-the-shelf product that does what I need.

Details:

Devices I use:

  • Desktop PC, dual-boot Windows 10 / Arch Linux
  • Laptop (Arch Linux)
  • NAS (self-build, FreeNAS running on it)

I've used rsync so far to backup to the NAS (one-way, my dual-boot setup uses the same storage for both OS).

The problem is, I would like to get the Laptop in the sync. Therefore I need to do some kind of two-way sync.

Problems that arise here:

  1. I often save files only seconds before the shutdown, so I need to monitor file changes instead of running a sync tool periodically

    • I cannot sync my laptop outside of the local network (and in the local network, I use the desktop PC most of the times); the local (updated) version could be quite out of date and give some conflicts.
  2. I do want to use the local network only (probably the existing network share of the NAS), not an ownCloud or something similar, because we have quite limited bandwidth where I live and I have some rather huge files to keep synced.

My ideas

I found this question: How do I setup rsync and any other scripts to mimic a minimal viable replica of DropBox?, and looked into a solution using lsyncd (Live Syncing Daemon), which synchronizes local directories with remote targets. This could solve problem #1, but I still need to be prepared for conflicts. Furthermore, I could not find any docs about lsyncd on Windows, so I assume it won't work there.

I also looked at the potential to incorporate Unison into a solution; it seems to do two-way syncs. Maybe I could use it together with lsyncd. I'm not sure about the conflict handling, though. I found a hint that with -batch, conflicts are just skipped. But how do I handle conflicted files in the long term (I want to sync them, after all)?

So my question is, how can I accomplish this?

  • 1
    You are asking an off-topic question (software shopping). Questions seeking product, service, or learning material recommendations are off-topic. See On Topic. Try softwarerecs.stackexchange.com but please first read What is required for a question to contain "enough information". – DavidPostill Oct 30 '16 at 22:33
  • 1
    Please be aware that the question is not only about a software recommendation, but rather about ideas how to solve the problem (possibly with unix tools). Thank you for the On Topic link, actually I was looking for something like this when I asked the question as I was not sure whether to put this here, but I could not find this one. I do not think this belongs to softwarerecs.stackexchange.com either, as I don't expect there even is a ready-to-use tool as stated in their OnTopic. I'll try to edit the question accordingly. – Qw3ry Oct 31 '16 at 12:11
  • 1
    @DavidPostill As stated in OnTopic, the question is now about "personal and home computer networking" and no more "asking for a product, service or learning material recommendation". I think this fits these suggestions quite well now. – Qw3ry Oct 31 '16 at 12:20
2

Depending on your NAS - I'm pretty fond of BitTorrent sync for this. It runs on most systems (including raspberry pis and mips systems) with support from quite a few consumer NAS systems (like my seagate). You just pick a folder to share, generate a key, and use the key to set up the share on each system you want it on. Its smart enough to detect you're on a LAN, works outside of it. The free version works well enough for my needs - I mainly use it to sync downloads from a system I keep on all the time and backups of many small files that I add to infrequently.

It'll do what you need - one way or two way sync, using the lan if needed. It deals with conflict by keeping a copy of deleted files, (which I turn off for my use cases) and generally runs well enough that I hardly notice it.

  • I'll have a look into it! Maybe better to use this solution instead of patching sth together by myself. The advantage of the scripting approach is (for me), that the SMB-share is automatically backed up (it's a zfs). OTOH the disadvantage is that my script might break with updates for unison... – Qw3ry Nov 16 '16 at 10:52
  • I'll accept your solution, as most ppl reading this will probably be better of with bitorrent sync. – Qw3ry Nov 16 '16 at 10:55
2

So, my self-developed solution requires a bit more scripting than the one of Journeyman Geek. I used unison for the sync-part and zenity to warn for conflicts.

Scripts

The bash script (put this into your startup-applications):

#!/bin/bash
if [ -z "$(arp | grep MAC_ADDRESS_OF_NAS)" ]; then
    zenity --warning --text="NAS not found. Aborting."
fi;
unison sync -batch=true -repeat=watch |& while read -r line; do
    if test -n "$(echo $line | grep skipped:)"; then
        zenity --warning --text="$line"; 
    fi;
done

The unison configuration (put this as sync.prf into ~/.unison/):

root=/home/my-username/the-data-i-want-to-backup/
root=/mnt/NAS/some/sub/path/

# follow all symlinks
follow = Path *

# automatically do everything (if there are no conflicts)
auto = true

# for some reason unison fails if I sync the permissions
perms = 0
dontchmod = true

Detailed explanation (and troubleshooting for derivates)

  1. First I check, if the NAS is found in the local network. If its MAC-address does not appear in the arp output, I assume I don't need to sync. This needs probably some further tweaking as I rarely turn my laptop off completely, so the script won't be started when I connect to the network. Maybe I'll add a cron job or some kind of "on-network-connect"
  2. zenity can be used to create graphical dialogs from a bash script. You probably need to install it. Basic usage is pretty straight forward.
  3. unison sync starts the synchronization. It looks for a configuration profile called ~/.unison/sync.prf. I could substitute sync by something else (both in the filename and the command).
  4. To skip conflicted files, I specify -batch=true. I don't do this in the config file because I want to run unison sync manually if a conflict happens.
  5. I don't need a third party tool to watch for changes: -repeat=watch does exactly what I want. Sidenote: actually it starts a full resync on any change, so it could do better, but that's ok for me.
  6. The output of the unison-command is a bit strange, took some time to get this working:
    • If there's a conflict, the output contains skipped: FILENAME
    • That seems to happen on STDERR rather than on STDOUT, so I need to pipe with |&
    • For some reason I could not figure out it did not work with a simple grep for me. I tried something like unison sync ... |& grep "skipped:" | xargs zenity --text="{}", but that would not execute until unison finally finished (it worked fine without -repeat=, and it worked fine without the grep)
  7. For above reasons I sticked with a while loop to read the input and use the grep only on each line itself
  8. In the unison profile, notice that I used absolute paths. For some reason it did not work with ~/my/folder

Additional notes

  • I'm someone who tends to over-engineer. I setup a script generating me the unison-profile. In case you're interested:

    #!/bin/bash
    
    HOME_DIR=$(cd ~; pwd)
    
    REPO_DIR="$(dirname "$(readlink -f $0)")"
    
    UNISON_PRF=$HOME_DIR/.unison/sync.prf
    
    # write sync script
    read -e -p "Local sync dir: [like ~/Data] " LOCAL_SYNC_DIR
    # make path absolute:
    LOCAL_SYNC_DIR="${LOCAL_SYNC_DIR/#\~/$HOME_DIR}"
    read -e -p "Remote sync dir: [like /mnt/NAS/Data/] " REMOTE_SYNC_DIR
    REMOTE_SYNC_DIR="${REMOTE_SYNC_DIR/#\~/$HOME_DIR}"
    
    rm "$UNISON_PRF"
    echo "root=$LOCAL_SYNC_DIR" > "$UNISON_PRF"
    echo "root=$REMOTE_SYNC_DIR" >> "$UNISON_PRF"
    cat "$REPO_DIR/sync.prf" >> "$UNISON_PRF"
    
    # user info
    echo "IMPORTANT: You need to put $REPO_DIR/unison/sync.sh to your startup applications for this to work properly!"
    

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.