0

I set up a Data Science Virtual Machine for Linux (Ubuntu) on Azure and want to check the installation of GPUs following these TensorFlow directions. The first command shows that a GPU is available with Tesla M60:

$ lspci | grep -i nvidia
db4d:00:00.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)

The second command fails with a cryptic message:

$ sudo docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:385: running prestart hook 1 caused \\\"error running hook: exit status 1, stdout: , stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods configure --ldconfig=@/sbin/ldconfig.real --device=all --compute --utility --require=cuda>=10.0 brand=tesla,driver>=384,driver<385 --pid=31149 /data/docker/overlay2/16e2b65fa0831681029432e3936005fa2796afd6d5a50c297d6bc0693e57a0b0/merged]\\\\nnvidia-container-cli: requirement error: unsatisfied condition: driver < 385\\\\n\\\"\"": unknown.

How can I set up a machine to run the Nvidia docker image?

0

This NVIDIA GitHub issue and this part of the error message:

--require=cuda>=10.0 brand=tesla,driver>=384,driver<385

suggest it is a driver issue. I don't quite understand why.

Solution using Docker, but without your image

The simplest solution is to use different Azure images: both NVIDIA GPU Cloud Image and NVIDIA GPU Cloud Image for Deep Learning and HPC will run that Docker image.

Solution using your image, but without Docker

Alternatively, you can still use Data Science Virtual Machine for Linux (Ubuntu) but without the containerisation of Docker. Conda, for example, can set up an environment (where the initial yes | answers yes to the prompts to install the packages):

yes | conda create -n TF python=2.7 scipy==1.0.0 tensorflow-gpu==1.8 Keras==2.1.3 pandas==0.22.0 numpy==1.14.0 matplotlib scikit-learn
export PATH=$PATH:/data/anaconda/envs/TF/bin
export PATH=$PATH:/data/anaconda/envs/py35/bin

These commands pull the official models from Tensorflow:

git clone https://github.com/tensorflow/models.git
export PYTHONPATH="$PYTHONPATH:./models"

A first call to nvidia-smi shows that the GPU has no running processes:

$ nvidia-smi
Mon Jan 21 16:26:02 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.44                 Driver Version: 396.44                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla M60           On   | 0000DB4D:00:00.0 Off |                  Off |
| N/A   39C    P8    14W / 150W |      0MiB /  8129MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

When you leave the official MNIST model running in the background for a little while, you will see one process using the GPU:

$ python models/official/mnist/mnist.py &
[1] 25967
$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.44                 Driver Version: 396.44                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla M60           On   | 0000DB4D:00:00.0 Off |                  Off |
| N/A   37C    P0    77W / 150W |   7851MiB /  8129MiB |     93%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     26077      C   python                                      7840MiB |
+-----------------------------------------------------------------------------+

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.