0

I have a cluster consisting of multiple nodes. Now I need to synchronize a folder on these nodes, and each node's folder may be added new files. Once a new file is added to the folder in one of the nodes, it needs to be synchronized to other nodes.

I want to know what is a good solution to this problem. I hope you can help me, thanks!

3
  • 1
    Did you have a look at rsync? Dec 14, 2020 at 7:41
  • Hi, Gerard! I saw examples of rsync today, most of which are syncing the folder of one node to another node. In my case, the files of all nodes may be changed, so I don't know whether the rsync can solve my problem: once the file changes of any node, and then it should be synchronized to other nodes.
    – wei wang
    Dec 14, 2020 at 12:25
  • 1
    “…once the file changes of any node, and then it should be synchronized to other nodes.” Sounds like you have a system architecture issue. The way you deal with this is to have a shared mounted storage device that all nodes use. Like something that is an NFS mount on each node that is located at /opt/node_storage/ for example. That way all nodes have access to the same content without having to jump through elaborate — and often unsatisfactory — sync loops. Dec 16, 2020 at 4:21

3 Answers 3

2

A well proven way is to use rsync.

For more than a few nodes it would be better to create a script (with node addresses/or names, list of folder synchronized and maybe a kind of parameters.

If a small delay in sync doesn't make a problem, than I would put a line in crontab to run the script in an interval (e. g. every hour). It's good to consider number of files, their size and time needed to copy new ones.

If the delay might be a problem, there are some better, but much more complicated solutions - network share, distributed filesystem etc.

A script example:

#! /bin/sh

#don't put our own ip/name here!

nodes="1.2.3.4 10.20.30.40 10.0.0.1"

folder="/data/work"

for node in $nodes ; do

rsync -aqW $folder/ $node:$folder

done
5
  • Hi, d.c.! Thank you for your reply! In my opinion, this script is to be run on all nodes. As long as the file in one node changes, the new file will be synchronized to other nodes. I don’t know if my understanding is right?
    – wei wang
    Dec 14, 2020 at 12:18
  • I saw examples of rsync today, most of which are syncing the folder of one node to another node. In my case, the files of all nodes may be changed, so I don't really understand whether the script you provide can realize the file changes of any node, and then it will be synchronized to other nodes.
    – wei wang
    Dec 14, 2020 at 12:23
  • Yes, the script is to be run on every node. The list nodes= would contain all other nodes except the local one. And yes, it will distribute files from every node to every other. However it will not work as full sync - files moved will be duplicated and files removed are likely to reappear. For better bandwidth utilization it would be better not to run the script on all machines at the same time.
    – d.c.
    Dec 14, 2020 at 16:01
  • Thank you for your reply! I basically figured out this problem. I can also understand why the deleted files came out again, but I didn't understand why the files moved will be duplicated. Can you tell me the reason?
    – wei wang
    Dec 15, 2020 at 0:50
  • That's just the consequence of the previous simple facts. If you have a file in folder/a and move it into a folder/b, the file will be replicated from folder/b to all other nodes. But the original place for the file (folder/a) will get back to the place from other nodes too. In reality multinode synchronization is somehow complicated and requires a kind of uuids for every item.
    – d.c.
    Dec 15, 2020 at 19:17
1

Let me suggest a NAS. As there is only one folder, no synchronization problem.

1

If the changes are usually conflict-free, syncthing might be useful.

Unlike a NAS, each machine is fully independent, but when you write multiple files with the same name on different machines, all but one of them are renamed (with a checksum appended to the file name) and you need to decide which ones to keep.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .