29

I'm looking for a way to reboot machines after getting an email notification.

The setup I want to use this on is a render farm. We have 5 boxes with 8 Nvidia GTX 980Ti in them. They're running Windows 10 Professional 64-bit.

The issue we are having is that they sometimes freeze or "stall" while rendering. I have it set up so once this happens I receive an email saying "x machine has stalled."

I have automated everything on the machines; they log in automatically, open up the necessary programs, initiate a connection with our license server, and map all the required network drives.

Basically all you have to do is turn it off and back on, and it works again.

So what I'm wondering is, can I set up a program, event, or a script that will tell the machines to reboot after I receive the email saying a machine has stalled?

The machine that will be running this is running Windows 7 Pro 64-bit.

  • 3
    Do you know what is stalling and how often ? A better plan maybe to make the computer restart daily avoiding the stall all together – Shaun07776 Apr 17 '16 at 16:53
  • @Shaun07776 Unfortunately that is the issue, it is random, i have sent log file after log file to each software vendor we are using trying to resolve this issue, as of right now we don't know what is causing them, or when they happen, which is why i am trying to automate everything i can in the meantime until we figure it out. – RyanIG Apr 17 '16 at 19:02
  • 23
    Why cant the job which detects the stall and sends the email just restart it ? Why does the email need to be the trigger ? – exussum Apr 18 '16 at 7:06
  • @exussum The program we are using is Called Deadline 7.2 it is meant to do this, i have configured it so the software does that, but it doesn't work for some reason, the program allows for python scripting through a web service but i can't get that to do much either. – RyanIG Apr 18 '16 at 9:47
  • @RyanIG - Just out of curiosity, does this need some special sort of shutdown procedure, the machine not responsive enough? – stefgosselin Apr 26 '16 at 4:17
44

You want a device called a "Remote Power Switch", one that we have used at my workplace to remotely restart cable modems is the UIS-322b below:

enter image description here

You can set it up with an IP address, and then log in to it to power on, power off, or power cycle either device connected, and set it on a schedule. You cannot control it via email, but through the web interface. You could still rig something up via wget or curl to automatically send a web request to it. It may have other features and control possibilities I'm unaware of.

If you are running rendering systems that consume a lot of power I am not sure if the specific device above can handle them, but there may be others out there that do. Another thing to look for is "Controllable Power Strip."

  • 15
    It basically remotelu unplugs and plugs in the device for you. – LawrenceC Apr 17 '16 at 12:57
  • 1
    There also exist similar devices designed for data center usage. Those are designed to be mounted in a rack and have a lot more output ports. Those I have seen were controlled over RS-232. That way you could attach them to a unit with lots of RS-232 ports such that this one unit would give you remove console and remote power cycling of all the individual servers in an entire rack. – kasperd Apr 17 '16 at 15:13
  • 1
    Some UPSs can provide this functionality too. You can connect to the UPS (via HTTP, or Telnet, or software released by the manufacturer) to connect via a cable (Ethernet, or a "serial cable"). Specific details vary based on individual solutions. Some UPSs only support this when using an add-on card (inserted in the UPS), purchased separately. I imagine that they are more expensive than the device LawrenceC is showing, because the UPSs also provide the feature of being battery backed up. However, the higher cost may be acceptable due to the inclusion of the extra useful feature. – TOOGAM Apr 17 '16 at 16:28
  • 1
    There is software that is part of the Red Hat Cluster Suite called fencing agents, which provides an abstract CLI interface to restarting these devices, including KVM as a "power fencing" device. It has been useful for me. – No Username Apr 18 '16 at 3:52
  • 4
    @RyanIG Maybe you can set one of these devices up so that they perform the hard reset automatically every 10 minutes or so unless they get a command via ethernet. Then you can hook the computers up to their own power supply and let them send the don't reboot me command every 3 minutes or so. If the computer hangs, it will also stop sending the command and get a hard reset. It's basically what's called a "watch dog (timer)" in embedded systems and it would bypass the whole email communication. – null Apr 18 '16 at 14:52
33

Can I setup a program, event, or a script that will tell the machines to reboot?

It depends on what you mean by:

they freeze or "stall"

If the remote machine has hung in a way that requires a "hard reset" then a remote restart will not work.

Having said that, you can use shutdown to remotely restart a device.

Example:

shutdown /r /f /m \\computer
  • /r - Shutdown and restart the computer.

  • /f - Force running applications to close without forewarning users.

  • /m \\computer - Specify the target computer.

You can run the above command from Outlook when you receive the appropriate email.

The following link How To: Trigger an application via email in Outlook should get you started.


shutdown usage

F:\test>shutdown /?
Usage: shutdown [/i | /l | /s | /r | /g | /a | /p | /h | /e] [/f]
    [/m \\computer][/t xxx][/d [p|u:]xx:yy [/c "comment"]]

    No args    Display help. This is the same as typing /?.
    /?         Display help. This is the same as not typing any options.
    /i         Display the graphical user interface (GUI).
               This must be the first option.
    /l         Log off. This cannot be used with /m or /d options.
    /s         Shutdown the computer.
    /r         Shutdown and restart the computer.
    /g         Shutdown and restart the computer. After the system is
               rebooted, restart any registered applications.
    /a         Abort a system shutdown.
               This can only be used during the time-out period.
    /p         Turn off the local computer with no time-out or warning.
               Can be used with /d and /f options.
    /h         Hibernate the local computer.
               Can be used with the /f option.
    /e         Document the reason for an unexpected shutdown of a computer.
    /m \\computer Specify the target computer.
    /t xxx     Set the time-out period before shutdown to xxx seconds.
               The valid range is 0-315360000 (10 years), with a default of 30.
               If the timeout period is greater than 0, the /f parameter is
               implied.
    /c "comment" Comment on the reason for the restart or shutdown.
               Maximum of 512 characters allowed.
    /f         Force running applications to close without forewarning users.
               The /f parameter is implied when a value greater than 0 is
               specified for the /t parameter.
    /d [p|u:]xx:yy  Provide the reason for the restart or shutdown.
               p indicates that the restart or shutdown is planned.
               u indicates that the reason is user defined.
               If neither p nor u is specified the restart or shutdown is
               unplanned.
               xx is the major reason number (positive integer less than 256).
               yy is the minor reason number (positive integer less than 65536).

Further Reading

  • 2
    Thanks for the fast response!When they "Freeze or Stall" sometimes the system is completely hung and it requires a hard reboot, other times you can get away with rebooting it via command line or through team-viewer. this trouble i'm having with figuring out how to link scripts to the email, would you have any threads or resources on how to do this? We are using the gmail but i could run it through outlook if required. cheers! – RyanIG Apr 17 '16 at 12:09
  • No worries, thanks for you help anyway! ill keep digging. – RyanIG Apr 17 '16 at 12:22
  • @RyanIG How To: Trigger an application via email in Outlook should give you a starting point. Answer updated. – DavidPostill Apr 17 '16 at 12:26
16

If you have actual servers, they likely have a remote management feature that can be connected to even if the machine's main OS is completely frozen. Generally, you use SSH to connect to the controller. The specific command issued depends on the brand of your server.

For a Dell servers, which use DRAC (Dell Remote Access Controller), the command to execute after SSH'ing into the DRAC is:

racadm serveraction hardreset

That performs a hard reset, equivalent to yanking out the power cord and then restarting the machine. If the machine isn't completely hung, powercycle instead of hardreset might do the job. (Source.)

HP servers use iLO, which is also accessible over SSH. The hard reset command is simply:

reset hard

(Source, the official PDF reference to iLO scripting.)

You can wire up the e-mail receiving to the execution of these commands with DavidPostill's answer. There are SSH clients for Windows; I believe PuTTY has a command-line interface.

  • Ill be taking a look into this as we use 3 dell powerblade servers, quick question though, does it have to be on the Domain controller or if we had a license server could it be setup on that? the machines are constantly connected to both. Thanks for your answer! – RyanIG Apr 17 '16 at 16:10
  • 1
    The DRAC should be accessible from anywhere on the same network. Therefore, any computer that can receive the e-mail and reach the DRAC should be able to handle the whole system. – Ben N Apr 17 '16 at 17:56
  • Awesome, ill look into this and let you know how i get on, cheers! – RyanIG Apr 17 '16 at 19:03
13

Use Outlook.

Link at the bottom of the answer has information that helped with this answer. Kudos to GuidingTech.com

Create a .bat file first.

shutdown.exe /r /t 00

Save as Restart.bat

Make sure the computer is logged in with the email you receive the email Notification in.

You can setup different accounts for each computer if you wish and have your email setup to forward emails your receive as notifications.

Example.

Computer1@email.com is the Host PC with a Graphic card that is not Rendering.

Testemail.com is your email you receive notifications on.

When the Email is received have your outlook auto forward the email back to the corresponding email. which in this case is Computer1@gmail.com

Computer1@email.com received at Testemail@email.com then have that email forward to the all Computers.

The Trick here is to setup a different Notification email to each computer so it knows which to shutdown.

For Setting up a Rule to Forward follow these instructions and enter in a word in the subject based on the pc you want to restart.

Note: This is for setting up your Computer which you receive the emails next we are going to setup the individual pcs.

Step 1. Now open MS Outlook. Under Home tab, go to Rules and expand the >menu. Select Manage Rules and Alerts. enter image description here

Step 2. On the Rules and Alerts window chick on New Rule while under Email >Rules tab. enter image description here

Step 3. Start off by selecting your template as Apply rule on messages I >receive. Click on Next. enter image description here

Step 4. Once again we need to specify which rule you want to use on the Receiving end which will be the Computer the forwarded email shows up at. This will be the rule for restarting the Computer you Specified.

A. With specific words in the subject– Another constraint so that the >receiving email is identified better.

B. With specific words in the body– Sometimes, only the subject may be >ambiguous. So a body makes it more refined.

Select them and hit next.

enter image description here

Now Specify what you want the email to look for to trigger the Restart.

Example if you choose subject.

Say the word you choose is Computer1-RenderIssue

You will type that in the "Specify for words or Phrases to search for in the subhect"

Now that you have the that setup its time to do the final step.

Step 5. Next comes the action to be applied when an email is received and >all the conditions match. Select start application and delete it. enter image description here

**Note:**If you do not delete it. If the computer is restarted it if outlook opens it will go in a loop.

Step 5a For starting application action, click on the link and select the file you created Restart.bat Click on Next.

Step 5b Click next on *"Are there any Exceptions?"

Step 6 Give the rule a name for future references, turn it on and click on >Finish.

Now whenever One of you computers have an issue when Rendering the Email will be sent to you with the Specified Computer. This will forward the email to that computer and restart the machine.

Hope this helps. Tried to format it the best I could while doing my Job.

Reference for some of the work here

  • Alright David I got what you meant its taken care of now. – NetworkKingPin Apr 18 '16 at 8:14
  • 2
    That's much better ;) – DavidPostill Apr 18 '16 at 8:18
6

To be honest, if all you want to do is to (hard) bounce a box, the easiest way is to "press the reset switch".

Since you need to do a HARD reset, you could either mess about on the High Voltage end (rather risky), or do it on the motherboard.

Some high-end motherboards have a built-in network card/controller for bouncing the PC. However you can also wire the reset-switch to a relay board and control that using a separate machine.

I would advise connecting a Raspberry Pi via the GPIO to a relay-board. Then attach each relay to the reset switch on the motherboards.

Raspberry Pi 2

Raspberry Pi Relay HAT

3

Have you considered running your server as a client on a virtual machine? You'd run the VM sofware (VMware, VirtualBox) on the physical server, then run a virtual machine with your rendering software as the client. With no other processing going on on the machine, you should get comparable performance, and it would be easier to stop and restart a virtual machine than a physical one.

  • 3
    I know where you are going with Virtual Machine concept but your statement about comparable performance doesn't make much sense. – pun Apr 18 '16 at 14:22
  • How so? You're basically just wrapping a physical server with a VM and running it on the same hardware. It's not like you're trying to run multiple VMs. According to this paper from 2009 "if you're using this year's VMware or Xen technologies, Intel Nehalems and Shanghai Opterons and the new i/o adapters -- virtualization has most performance issues pretty much beat." I don't imagine virtualization has backslid much since then. – TMN Apr 18 '16 at 15:21
  • 1
    Forgot to mention: this is a rendering farm, so most of the work is being done by the graphics cards, and their performance won't be degraded by running the host in a VM. – TMN Apr 19 '16 at 13:47
  • 1
    VM access to GPU's can be tricky (very VM specific, very card specific, if supported at all), so verify that it works for you. – toddkaufmann Apr 20 '16 at 1:41
1

I would suggest you investigate whether the mainboards of these machines have a watchdog timer on board. I believe that these are reasonably common at least with Intel chipsets. If watchdog hardware is present, find suitable Windows software to set up the timer, and reset it on a regular basis as long as the machine remains unfrozen. If the machine freezes, the software won't be able to reset the timer any more, and the watchdog will reset the machine. That should at least resolve the hard system freezes.

Depending on the software, you may be able to add additional requirements for the timer reset, like network connectivity, render process still running, render process made some progress since last timer reset, or similar. I don't know available software on Windows, I've only been using watchdogs on Linux so far.

-1

One creative solution is described here

The boys solved similar problem by 2 PC standing against each other. When email came, the computer opened (and then closed) CD trailer, so it pressed reset button on the other PC.

Of course you can use anything other, which is able to get signaled and move any part there and back (like servo connected to something - parallel port of old PC, GPIO from Rapsberry, or Arduino, or just anything else you have around)

  • 1
    Fun reading, but please include the important parts in your post - answers should be as self-contained as possible. – Ben N Apr 19 '16 at 12:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.