0

Originally, I just wanted to synchronize some selected files of my 1TB OneDrive to my Notebook SSD and all my files to my external HDD. Cloud storage programs (dropbox, gdrive, onedrive, resilio, syncthing...) seem to dislike two sync'ed folders with the same content on the same device, so either one has to trick them with a virtual machine, or look for another solution.


I thought a bit more about the ultimate storage solution (at least for my use case) and it got a bit more complicated, but it is a rather normal setup for many people nowadays:

  • notebooks with small drives
  • smartphones & tablets
  • external HDDs
  • (NAS)
  • (cloud storage)
  • SD-card (e.g. from camera)

Aggregation (virtual drive of all available data)

On each storage device, I want to have a virtual home drive where all my files from all devices are aggregated. The current, active documents, photos, videos are usually available synchronized on all devices for organizing, editing, deleting, etc. Old bloat, large video files, backups might only reside in one or two locations with sufficient storage space (NAS, exHDD), but are still shown in my home drive and readily available, if connection is sufficient. Such an abstraction layer that aggregates the data from multiple locations to a single one could be possibly realized with symbolic links (mklink), but I guess it would require quite some administration overhead.

Synchronization (decentralized, selective)

Next, the different data sources need to be synchronized (file table always, files/folders selectively) as soon as two devices can communicate (SSD <-USB-> exHDD // smartphone <-WiFi-> NAS // exHDD <-USB & Internet-> cloud). I guess this could be achieved with syncthing, or maybe even start with git (for added version control). Another option would be nextcloud or seafile, but they are both centralized approaches, which I'd like to avoid. I could imagine blockchain storage like Sia or Storj to be also good contenders, but would like to hear what the community has to say. Resilio Sync is perfect for the decentralized sync part, but since it's closed source, I think it will be hard to combine it with the aggregation part.

TL;DR

  • aggregate all available storage (SSD, exHDD, NAS, cloud, smartphone, ...) to one virtual folder (symlink, mklink)
  • synchronize decentralized between locations (syncthing, git)

  • in other words, I want a network folder with Windows offline-files and slow-link capabilities, but in a decentralized network infrastructure

What is the status quo? Have I overseen viable solutions? Am I asking for the impossible?

8
  • I don't think there is a status quo. I think you find and use a set or tools that work for your needs and stick with them. I use Resilio Sync, Linux with MDADM, Samba, NFS, and folder sync and it works to provide everything you ask. But your setup and skill set is different so the solution for you might look different. There are lots of ways to "skin" things, you just have to find YOUR way.
    – Damon
    Mar 9, 2018 at 4:37
  • @Damon thanks for your pointing me in the right direction. Do you have any tutorials or other resources that would help in getting this set up? Do you have this process automated? And do you have mixed folders (P:\folder1 on SSD (fastest), NAS & cloud, P:\folder2\ on NAS (fastest) & cloud, P:\folder3\ on cloud, P:\folder4\ on exHDD (fastest) and NAS) Resilio unfortunately can't sync two locations on single device.
    – aXeL-HH
    Mar 9, 2018 at 18:56
  • The only proprietary things I use are Resilio & Folder Sync, the rest is vanilla stuff. Folder sync is scheduled and automated, Resilio just runs as a service and is always "on". I'm not sure what you mean by Resilio cant sync two locations; I have it sync all sorts of things; personal folders, app folders with custom settings, mobile folders, etc. Admittedly I run VMs so I have technically have 2 different Resilio instances to tie into, business vs home. I will run Resiio in a VPS soon to provide better upload capacity. I run cheap ol RAID10 on HDD; only SSDs for OS on workstations.
    – Damon
    Mar 10, 2018 at 0:50
  • I would say draw your infrastructure on paper including what services will be ran where and just fill in with apps and services as needed. I don't have any one tutorial. Just google each individual piece. Every thing I use is "free" although I did pay for Resilio Sync pro and folder sync, but you do not have to. It has taken me years to implement, but I started with a plan on paper and just updated it as I went along and learned new things. Just start at somethin' and stick with it!
    – Damon
    Mar 10, 2018 at 0:56
  • reg. sync of 2 local folders: Resilio, Dropbox, OneDrive etc. won't sync c:\folder1 and d:\folder1 (SSD & exHDD or NAS). I understand, their protocol is inefficient due to communication overhead for secure internet sync, and dropbox LAN sync is actually pretty fast! But I have a very bad feeling using different tools doing pretty much the same task, esp. automated syncing, as that usually results in chaos if not done in a very precise order. I don't see the user's systems as entities, but the user's storage devices, which need to be in (selective) sync.
    – aXeL-HH
    Mar 11, 2018 at 10:27

0

You must log in to answer this question.

Browse other questions tagged .