This NVIDIA GitHub issue and this part of the error message:
--require=cuda>=10.0 brand=tesla,driver>=384,driver<385
suggest it is a driver issue. I don't quite understand why.
Solution using Docker, but without your image
The simplest solution is to use different Azure images: both NVIDIA GPU Cloud Image
and NVIDIA GPU Cloud Image for Deep Learning and HPC
will run that Docker image.
Solution using your image, but without Docker
Alternatively, you can still use Data Science Virtual Machine for Linux (Ubuntu)
but without the containerisation of Docker. Conda, for example, can set up an environment (where the initial yes |
answers yes to the prompts to install the packages):
yes | conda create -n TF python=2.7 scipy==1.0.0 tensorflow-gpu==1.8 Keras==2.1.3 pandas==0.22.0 numpy==1.14.0 matplotlib scikit-learn
export PATH=$PATH:/data/anaconda/envs/TF/bin
export PATH=$PATH:/data/anaconda/envs/py35/bin
These commands pull the official models from Tensorflow:
git clone https://github.com/tensorflow/models.git
export PYTHONPATH="$PYTHONPATH:./models"
A first call to nvidia-smi
shows that the GPU has no running processes:
$ nvidia-smi
Mon Jan 21 16:26:02 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.44 Driver Version: 396.44 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla M60 On | 0000DB4D:00:00.0 Off | Off |
| N/A 39C P8 14W / 150W | 0MiB / 8129MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
When you leave the official MNIST model running in the background for a little while, you will see one process using the GPU:
$ python models/official/mnist/mnist.py &
[1] 25967
$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.44 Driver Version: 396.44 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla M60 On | 0000DB4D:00:00.0 Off | Off |
| N/A 37C P0 77W / 150W | 7851MiB / 8129MiB | 93% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 26077 C python 7840MiB |
+-----------------------------------------------------------------------------+