Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have a cluster that I manage and from time to time I get emails from each node (and head node) begging to be restarted after an automatic upgrade.

Currently, my best solution so far is a shell script like:

 $> cat cluster_reboot.sh
 ssh root@node1.host.edu reboot
 ssh root@node2.host.edu reboot
 ssh root@node3.host.edu reboot
 ssh root@node4.host.edu reboot
 ssh root@node5.host.edu reboot
 ssh root@headnode.host.edu reboot

I end up just typing the root password six times, but it works, I guess. Is there a better way? Can I force the head node to reboot the computers for me?

UPDATE: I'd like to shy away from using keyless login for the root user... but that is a method that would definitely work.

UPDATE 2: Key pairs might have a use after all. What about a keyed login to the head node, then keyless login from the head node to all the compute nodes. Then something like the following:

 $> ssh root@headnode
 Enter password for 'root':
 [<headnode>]$: cat cluster_reboot.sh
 ssh root@node1.host.edu reboot
 ssh root@node2.host.edu reboot
 ssh root@node3.host.edu reboot
 ssh root@node4.host.edu reboot
 ssh root@node5.host.edu reboot
 echo "Nodes rebooted. Rebooting this computer now."
 reboot
 [<headnode>]$: sh cluster_reboot.sh
 "Nodes rebooted. Rebooting this computer now."

I think that might be the secure+easy solution.

share|improve this question
2  
Cycle power to the building? –  Phoshi Mar 23 '10 at 19:20

6 Answers 6

up vote 1 down vote accepted

Perhaps doing a check-and-see sort of system might work here for you as well?

Assuming you have some sort of HTTP server running on the head node, you could write a little php/perl/whatever script that lets you flag which computers are ok to reboot.

Then, on each node, put a cron job that polls that script to see if it's ok to reboot. If the head node sends back the ok, then it should reboot. Once the headnode has sent the ok-to-reboot command, it could clear the flag.

I would set this up using a directory with some text files in it, if the script sees the name then sends back the ok to reboot then deletes the file. Easy for you to manage because you just have to touch them to flag an ok. Secure because you don't ever have to pass any of that info directly to the filesystem.

Each script on the sub-nodes would just check by hostname or something;

http://headnode/reboot?name=subnode1

Checking every 5 minutes or so shouldn't be too taxing.

share|improve this answer

Configure SSH to use certificate based authentication. Then use a command like this:

$ for h in node1 node2 node3 node4 node5 headnode; do
>   ssh root@$h reboot
>   done
share|improve this answer

I'd suggest setting up out-of-band management for the cluster. This way, you can reach a console even if the box falls off the network, and remotely rebooting the machines is a snap. You'll need to spend money to purchase either a dedicated box with extra serial ports and roll your own, or a purpose built serial port concentrator. This obviously isn't the cheapest solution, but it will make remote management quite a bit easier.

share|improve this answer
    
Good solution. But not in my budget of $0! –  physicsmichael Mar 24 '10 at 4:27

Use ssh key pairs so you don't have to type the password so many times.

share|improve this answer

Not a resolution for this problem, but Rocks is a cluster "distro" built on CentOS which has ssh keys and keeps tracks of how many nodes you have. You just run rocks run host reboot to reboot all your nodes. Rocks has all you need for a full fledged HPC cluster and I've personally tried it on a cluster with over 1200 nodes.

share|improve this answer
    
ordinarily i'd suggest posting something like this as a comment, but i think this is a fair answer. it's possible (maybe even likely) that the clustering distro the OP uses has a similar command. of course what command depends on the particulars of what the OP is using. thanks for bringing this up. –  quack quixote Mar 23 '10 at 20:37
    
This was useful. I'm putting together a different cluster with CentOS and now I'll look into Rocks for that one. –  physicsmichael Mar 23 '10 at 22:18

Use the ansible cluster administration tool, perhaps combined with the 'at' command to schedule the reboot to your liking.

You could set up a script ('playbook' in ansible terminology), but I generally just do it using the 'ad hoc' command line interface, something like this:

ansible clusternodes -m shell -a "echo shutdown -r +15 | at 6am Monday" -sK

Where clusternodes is an alias to a set of hosts defined in my ansible inventory, -m shell means 'run this on the command line', -a "blah" gives the shell command, and -sK means 'run in sudo mode and ask me for the password.

(Note: You will want to set up SSH keys giving you password-less access to your nodes first.)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.