On a GPU in Linux, how can a process be killed when another process is launched, and then restarted when the other process completes?

Question

I have a laptop that is normally running GPU cryptocurrency mining. I want the laptop to kill this cryptocurrency mining when any intensive deep learning process is launched. So, I'm not trying to lower the priority of the GPU cryptocurrency mining; I'm actually trying to kill the process when some obviously intensive GPU process (e.g. using TensorFlow) is launched.

How might this be done? Should I have a daemon running that watches for specific processes? It feels like this might be functionality already offered in Linux.

Answer 1

I would suggest using a systemd unit for launching and keeping alive a script that periodically checks for your machine learning process.

Place the following script in /usr/local/bin and make it executable with chmod +x

Script:

#!/bin/bash

#replace with your process names
plearning="tensorflow"
pmining="mining"

sleepint="30" #interval in seconds between checks

while true;
do
    if pgrep -x "$plearning" > /dev/null
    then
        killall -q "$pmining"
        logger "miner killed"

    fi
    sleep $sleepint
done

Create a systemd unit in /etc/systemd/system, for example miningswitch.service

Unit:

[Unit]
Description=test

[Service]
ExecStart=/usr/local/bin/miningswitch
ExecStop=killall miningswitch

Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

Start and enable the service with systemctl enable --now miningswitch.service

Answer 2

As the GPU relies on a CPU process and the hard disk for its data, you could perhaps use nice together with ionice to reduce both the CPU and I/O priorities of the mining process.

Otherwise, you could create a cron task, running periodically, to check for the presence of the deep learning process and kill the miner.

This task can be as simple as:

#!/bin/bash
if pgrep -x "deep-learning-process" > /dev/null
then
    kill "mining-process"
fi