I created a glusterfs (6.1) cluster with 6 servers (node01 - node06) in Replicate (=6) mode but cannot recover a failed server (node01, this vm was reset/rebuilt but I want to keep its hostname/ip) following this tutorial (https://support.rackspace.com/how-to/recover-from-a-failed-server-in-a-glusterfs-array/):

  • I can add node01 back to the cluster by getting UUID from a running server (gluster peer status) and updating "/var/lib/glusterd/glusterd.info" file (node01)

  • I then updated extended attributes (trusted.glusterfs.volume-id) of node01 directory and restarted gluster service:

sudo setfattr -n trusted.glusterfs.volume-id -v hash /data/brick1/vol

sudo systemctl restart glusterd

sudo gluster volume heal vol full

sudo gluster volume heal vol info

  • logs and outputs:

sudo gluster volume status

Brick node01:/data/brick1/volr              N/A       N/A        N       N/A
Brick node02:/data/brick1/volr              49152     0          Y       5734
Brick node03:/data/brick1/volr              49152     0          Y       5733
Brick node04:/data/brick1/volr              49152     0          Y       5691  
Brick node05:/data/brick1/volr              49152     0          Y       5721  
Brick node06:/data/brick1/volr              49152     0          Y       5723

sudo gluster volume heal vol full

Launching heal operation to perform full self heal on volume volr has been successful
Use heal info commands to check status.

sudo gluster volume heal vol info

Brick node01:/data/brick1/volr
Status: Transport endpoint is not connected
Number of entries: -

Brick node02:/data/brick1/volr
Status: Connected
Number of entries: 0

Brick node03:/data/brick1/volr
Status: Connected
Number of entries: 0

Brick node04:/data/brick1/volr
Status: Connected
Number of entries: 0

Brick node05:/data/brick1/volr
Status: Connected
Number of entries: 0

Brick node06:/data/brick1/volr
Status: Connected
Number of entries: 0

Any help would be greatly appreciated.

  • 1
    What do you see in the log files at /var/log/glusterfs/, especially /var/log/glusterfs/glusterd.log? You have a good chance of finding the cause in that log file. – Deltik Apr 23 at 7:34
  • Nothing special,no error messages (E) found in glusterd.log. but in glustershd.log, I can see something like "0-vol-client-0: intentional socket shutdown(11)" – BrianX Apr 23 at 7:41
  • Is glusterd actually running on node01? You can check with sudo systemctl status glusterd.service. If it's not running or encountered some kind of error, maybe the cause is logged in sudo journalctl -xe --unit glusterd.service. – Deltik Apr 23 at 7:45
  • yes, it is running: sudo systemctl status glusterd, active: active (running). No error found in logs so far. – BrianX Apr 23 at 7:52
  • Is node01 peered with the other nodes? Check with sudo gluster peer status on node01 and any of the other nodes. – Deltik Apr 23 at 8:01

I was able to reproduce your problem with the exact same symptoms.

You correctly peered the reinstalled node, node01, but the brick replacement procedure isn't well-documented.

The official documentation has some weird instructions for changing out the faulty brick for a new one with a different name. It's unnecessarily complicated, and I think you want to use the same brick name.

You weren't able to get the brick running because the replacement brick didn't have necessary metadata stored in node1:/data/brick1/volr/.glusterfs.

Here's what I did to replace the brick:

Assuming that:

  • node01 is the rebuilt server that has already been peered,
  • there is a volume called vol, and
  • the brick to be replaced is node01:/data/brick1/volr

Do the following:

  1. On node01, shut down GlusterFS:

    sudo killall -9 glusterd glusterfsd glusterfs
  2. Remove the brick to have a clean slate:

    sudo rm -rfv /data/brick1/volr
  3. Create the brick folder again:

    sudo mkdir -pv /data/brick1/volr
  4. Restart glusterd.service:

    sudo systemctl restart glusterd.service
  5. Reset the brick:

    sudo gluster volume reset-brick vol node01:/data/brick1/volr node01:/data/brick1/volr commit
  6. If the healing process has not started already, start it:

    sudo gluster volume heal vol full

Additional Resources

  • Haven't tried above cmds but just wondering does it work on a replica=3 cluster (we have to make sure brick count should be a multiple of the replica count) or even with a distributed dispersed cluster? – BrianX Apr 23 at 9:31
  • @BrianX: I have updated my answer with a better procedure that should work for all types of volumes. (I've only tested the instructions on a replica 3 volume.) – Deltik Apr 23 at 9:42
  • cool, it does work on my current setup. Will also try it out on a distributed dispersed cluster later. Many thanks! – BrianX Apr 23 at 10:07
  • @BrianX: Great! Please check the gray check box next to the answer to indicate that it was helpful. – Deltik Apr 23 at 10:15
  • Sure. Already did it. – BrianX Apr 24 at 4:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.