Is it possible to restart a computer in response to an email notification?

Question

I'm looking for a way to reboot machines after getting an email notification.

The setup I want to use this on is a render farm. We have 5 boxes with 8 Nvidia GTX 980Ti in them. They're running Windows 10 Professional 64-bit.

The issue we are having is that they sometimes freeze or "stall" while rendering. I have it set up so once this happens I receive an email saying "x machine has stalled."

I have automated everything on the machines; they log in automatically, open up the necessary programs, initiate a connection with our license server, and map all the required network drives.

Basically all you have to do is turn it off and back on, and it works again.

So what I'm wondering is, can I set up a program, event, or a script that will tell the machines to reboot after I receive the email saying a machine has stalled?

The machine that will be running this is running Windows 7 Pro 64-bit.

Answer 1

You want a device called a "Remote Power Switch", one that we have used at my workplace to remotely restart cable modems is the UIS-322b below:

You can set it up with an IP address, and then log in to it to power on, power off, or power cycle either device connected, and set it on a schedule. You cannot control it via email, but through the web interface. You could still rig something up via wget or curl to automatically send a web request to it. It may have other features and control possibilities I'm unaware of.

If you are running rendering systems that consume a lot of power I am not sure if the specific device above can handle them, but there may be others out there that do. Another thing to look for is "Controllable Power Strip."

Answer 2

Can I setup a program, event, or a script that will tell the machines to reboot?

It depends on what you mean by:

they freeze or "stall"

If the remote machine has hung in a way that requires a "hard reset" then a remote restart will not work.

Having said that, you can use shutdown to remotely restart a device.

Example:

shutdown /r /f /m \\computer

• /r - Shutdown and restart the computer.

• /f - Force running applications to close without forewarning users.

• /m \\computer - Specify the target computer.

You can run the above command from Outlook when you receive the appropriate email.

The following link How To: Trigger an application via email in Outlook should get you started.

---

shutdown usage

F:\test>shutdown /?
Usage: shutdown [/i | /l | /s | /r | /g | /a | /p | /h | /e] [/f]
    [/m \\computer][/t xxx][/d [p|u:]xx:yy [/c "comment"]]

    No args    Display help. This is the same as typing /?.
    /?         Display help. This is the same as not typing any options.
    /i         Display the graphical user interface (GUI).
               This must be the first option.
    /l         Log off. This cannot be used with /m or /d options.
    /s         Shutdown the computer.
    /r         Shutdown and restart the computer.
    /g         Shutdown and restart the computer. After the system is
               rebooted, restart any registered applications.
    /a         Abort a system shutdown.
               This can only be used during the time-out period.
    /p         Turn off the local computer with no time-out or warning.
               Can be used with /d and /f options.
    /h         Hibernate the local computer.
               Can be used with the /f option.
    /e         Document the reason for an unexpected shutdown of a computer.
    /m \\computer Specify the target computer.
    /t xxx     Set the time-out period before shutdown to xxx seconds.
               The valid range is 0-315360000 (10 years), with a default of 30.
               If the timeout period is greater than 0, the /f parameter is
               implied.
    /c "comment" Comment on the reason for the restart or shutdown.
               Maximum of 512 characters allowed.
    /f         Force running applications to close without forewarning users.
               The /f parameter is implied when a value greater than 0 is
               specified for the /t parameter.
    /d [p|u:]xx:yy  Provide the reason for the restart or shutdown.
               p indicates that the restart or shutdown is planned.
               u indicates that the reason is user defined.
               If neither p nor u is specified the restart or shutdown is
               unplanned.
               xx is the major reason number (positive integer less than 256).
               yy is the minor reason number (positive integer less than 65536).

---

Further Reading

• An A-Z Index of the Windows CMD command line - An excellent reference for all things Windows cmd line related.
• shutdown - Shutdown the computer.

Answer 3

If you have actual servers, they likely have a remote management feature that can be connected to even if the machine's main OS is completely frozen. Generally, you use SSH to connect to the controller. The specific command issued depends on the brand of your server.

For a Dell servers, which use DRAC (Dell Remote Access Controller), the command to execute after SSH'ing into the DRAC is:

racadm serveraction hardreset

That performs a hard reset, equivalent to yanking out the power cord and then restarting the machine. If the machine isn't completely hung, powercycle instead of hardreset might do the job. (Source.)

HP servers use iLO, which is also accessible over SSH. The hard reset command is simply:

reset hard

(Source, the official PDF reference to iLO scripting.)

You can wire up the e-mail receiving to the execution of these commands with DavidPostill's answer. There are SSH clients for Windows; I believe PuTTY has a command-line interface.

Answer 4

Use Outlook.

Link at the bottom of the answer has information that helped with this answer. Kudos to GuidingTech.com

Create a .bat file first.

shutdown.exe /r /t 00

Save as Restart.bat

Make sure the computer is logged in with the email you receive the email Notification in.

You can setup different accounts for each computer if you wish and have your email setup to forward emails your receive as notifications.

Example.

[email protected] is the Host PC with a Graphic card that is not Rendering.

Testemail.com is your email you receive notifications on.

When the Email is received have your outlook auto forward the email back to the corresponding email. which in this case is [email protected]

[email protected] received at [email protected] then have that email forward to the all Computers.

The Trick here is to setup a different Notification email to each computer so it knows which to shutdown.

For Setting up a Rule to Forward follow these instructions and enter in a word in the subject based on the pc you want to restart.

Note: This is for setting up your Computer which you receive the emails next we are going to setup the individual pcs.

Step 1. Now open MS Outlook. Under Home tab, go to Rules and expand the >menu. Select Manage Rules and Alerts.

Step 2. On the Rules and Alerts window chick on New Rule while under Email >Rules tab.

Step 3. Start off by selecting your template as Apply rule on messages I >receive. Click on Next.

Step 4. Once again we need to specify which rule you want to use on the Receiving end which will be the Computer the forwarded email shows up at. This will be the rule for restarting the Computer you Specified.

A. With specific words in the subject– Another constraint so that the >receiving email is identified better.

B. With specific words in the body– Sometimes, only the subject may be >ambiguous. So a body makes it more refined.

Select them and hit next.

Now Specify what you want the email to look for to trigger the Restart.

Example if you choose subject.

Say the word you choose is Computer1-RenderIssue

You will type that in the "Specify for words or Phrases to search for in the subhect"

Now that you have the that setup its time to do the final step.

Step 5. Next comes the action to be applied when an email is received and >all the conditions match. Select start application and delete it.

**Note:**If you do not delete it. If the computer is restarted it if outlook opens it will go in a loop.

Step 5a For starting application action, click on the link and select the file you created Restart.bat Click on Next.

Step 5b Click next on *"Are there any Exceptions?"

Step 6 Give the rule a name for future references, turn it on and click on >Finish.

Now whenever One of you computers have an issue when Rendering the Email will be sent to you with the Specified Computer. This will forward the email to that computer and restart the machine.

Hope this helps. Tried to format it the best I could while doing my Job.

Reference for some of the work here

Answer 5

To be honest, if all you want to do is to (hard) bounce a box, the easiest way is to "press the reset switch".

Since you need to do a HARD reset, you could either mess about on the High Voltage end (rather risky), or do it on the motherboard.

Some high-end motherboards have a built-in network card/controller for bouncing the PC. However you can also wire the reset-switch to a relay board and control that using a separate machine.

I would advise connecting a Raspberry Pi via the GPIO to a relay-board. Then attach each relay to the reset switch on the motherboards.

Answer 6

Have you considered running your server as a client on a virtual machine? You'd run the VM sofware (VMware, VirtualBox) on the physical server, then run a virtual machine with your rendering software as the client. With no other processing going on on the machine, you should get comparable performance, and it would be easier to stop and restart a virtual machine than a physical one.

Answer 7

I would suggest you investigate whether the mainboards of these machines have a watchdog timer on board. I believe that these are reasonably common at least with Intel chipsets. If watchdog hardware is present, find suitable Windows software to set up the timer, and reset it on a regular basis as long as the machine remains unfrozen. If the machine freezes, the software won't be able to reset the timer any more, and the watchdog will reset the machine. That should at least resolve the hard system freezes.

Depending on the software, you may be able to add additional requirements for the timer reset, like network connectivity, render process still running, render process made some progress since last timer reset, or similar. I don't know available software on Windows, I've only been using watchdogs on Linux so far.

Answer 8

One creative solution is described here

The boys solved similar problem by 2 PC standing against each other. When email came, the computer opened (and then closed) CD trailer, so it pressed reset button on the other PC.

Of course you can use anything other, which is able to get signaled and move any part there and back (like servo connected to something - parallel port of old PC, GPIO from Rapsberry, or Arduino, or just anything else you have around)