Good ways to restart all the computers in a remote cluster?

Question

I have a cluster that I manage and from time to time I get emails from each node (and head node) begging to be restarted after an automatic upgrade.

Currently, my best solution so far is a shell script like:

 $> cat cluster_reboot.sh
 ssh root@node1.host.edu reboot
 ssh root@node2.host.edu reboot
 ssh root@node3.host.edu reboot
 ssh root@node4.host.edu reboot
 ssh root@node5.host.edu reboot
 ssh root@headnode.host.edu reboot

I end up just typing the root password six times, but it works, I guess. Is there a better way? Can I force the head node to reboot the computers for me?

UPDATE: I'd like to shy away from using keyless login for the root user... but that is a method that would definitely work.

UPDATE 2: Key pairs might have a use after all. What about a keyed login to the head node, then keyless login from the head node to all the compute nodes. Then something like the following:

 $> ssh root@headnode
 Enter password for 'root':
 [<headnode>]$: cat cluster_reboot.sh
 ssh root@node1.host.edu reboot
 ssh root@node2.host.edu reboot
 ssh root@node3.host.edu reboot
 ssh root@node4.host.edu reboot
 ssh root@node5.host.edu reboot
 echo "Nodes rebooted. Rebooting this computer now."
 reboot
 [<headnode>]$: sh cluster_reboot.sh
 "Nodes rebooted. Rebooting this computer now."

I think that might be the secure+easy solution.

Answer 1

Perhaps doing a check-and-see sort of system might work here for you as well?

Assuming you have some sort of HTTP server running on the head node, you could write a little php/perl/whatever script that lets you flag which computers are ok to reboot.

Then, on each node, put a cron job that polls that script to see if it's ok to reboot. If the head node sends back the ok, then it should reboot. Once the headnode has sent the ok-to-reboot command, it could clear the flag.

I would set this up using a directory with some text files in it, if the script sees the name then sends back the ok to reboot then deletes the file. Easy for you to manage because you just have to touch them to flag an ok. Secure because you don't ever have to pass any of that info directly to the filesystem.

Each script on the sub-nodes would just check by hostname or something;

http://headnode/reboot?name=subnode1

Checking every 5 minutes or so shouldn't be too taxing.

Answer 2

Configure SSH to use certificate based authentication. Then use a command like this:

$ for h in node1 node2 node3 node4 node5 headnode; do
>   ssh root@$h reboot
>   done

Answer 3

I'd suggest setting up out-of-band management for the cluster. This way, you can reach a console even if the box falls off the network, and remotely rebooting the machines is a snap. You'll need to spend money to purchase either a dedicated box with extra serial ports and roll your own, or a purpose built serial port concentrator. This obviously isn't the cheapest solution, but it will make remote management quite a bit easier.

Answer 4

Use ssh key pairs so you don't have to type the password so many times.

Answer 5

Not a resolution for this problem, but Rocks is a cluster "distro" built on CentOS which has ssh keys and keeps tracks of how many nodes you have. You just run rocks run host reboot to reboot all your nodes. Rocks has all you need for a full fledged HPC cluster and I've personally tried it on a cluster with over 1200 nodes.