How to fix “command not found” and missing env entries in ssh terminals

Question

Desired behavior

When connected via ssh, "pip" (and other tools) should be available, just like they are in a "normal" terminal.

Also all the environment variables that are present in normal terminal sessions should be available in a ssh session (or in a shell opened via ssh).

Current behavior

Trying to execute "pip" gives a "command not found" error when connected through ssh, but the command works properly when executed in a normal terminal.

Setup

The ssh daemon is running in a docker container based on the image nvcr.io/nvidia/pytorch:20.08-py3. It is exposed as port 30022.

Opening /bin/bash as an interactive terminal into the container works as expected with all the tools available and environment variables set. The user inside the container is the default (root) user - same when connecting through ssh.

The ssh connection is done using a rsa keypair with the pubkey added in the container's /root/.ssh/authorized_keys.

Things tried

Connecting with ssh -p 30022 root@localhost, or ssh -p 30022 root@localhost -t "bash -l" didn't give a shell with the desired behavior.

Executing su - in a ssh session didn't help.

Doing source /root/.bashrc in a ssh session didn't help either.

Answer 1

Solved

Solution based on https://stackoverflow.com/questions/34630571/docker-env-variables-not-set-while-log-via-shell

For anyone having the same problem:

This problem was docker specific. The environment variables are set in the Dockerfile nvidia's image is built from. To restore the environment variables in a ssh session, I added RUN env > /etc/environment to my Dockerfile that's using the nvidia image as a base.

The file is read and used to populate the environment on ssh sessions as well, which also solves the missing $PATH entries responsible for the "command not found" error when calling pip or other tools.