Avoiding "nested sessions" warning when using tmux from systemd

Question

I want to use a single script to launch 2 sessions, a daemon session and a user session. I want the script to start upon boot without any users logged in.

The script I created works when launched directly, but only partially when started with systemctl start daemon.service as root (likewise at boot).

Essentially the script does this:

# Clean up any old tmux sessions
tmux kill-session -t daemon > /dev/null 2>&1 
tmux kill-session -t user > /dev/null 2>&1 
rm -rf /tmp/tmux-`id -u`

tmux new-session -d -s daemon
tmux send-keys "$DAEMON" C-m

# Start the main tmux session from which we'll create 
#  all window panes 
export TMUX=
export TERM=xterm
tmux new-session -d -s user
tmux list-sessions >> $LOG

# Various window setup using "tmux split-window -h" 
#  or "tmux split-window -v" - no other args

# Window panes created. Now wait for daemon process to open socket, then
echo "Daemon is now listening." >> $LOG

tmux send-keys -t 1 "$CMD1" C-m
echo "Sent $CMD1 to pane 1" >> $LOG
tmux send-keys -t 1 "$CMD2" C-m
echo "Sent $CMD2 to pane 2" >> $LOG
...

# Spin in a loop until the daemon process stops listening, then exit

Thats it. Simple. No nested sessions, tho tmux warns nonetheless. Why? I read elsewhere that setting the TERM and unsetting the TMUX env vars is necessary for a systemd process as there is no tty for tmux to use. It did seem to help, tho I've gone thru so many trials couldn't give you any details.

The symptom is both sessions start, the daemon looks normal but the user session panes are empty, but all panes created properly. The send-keys don't appear to be sent to them, tho the log looks perfect, no hangs anywhere.

I need this to work on different versions of tmux from 1.9 to 2.1 (ubuntu 16.04 & Debian 8.7 & 8.8). The user portion starts a "less" pager to view the daemon log, and 2 processes that can interact with a user. I put a "tmux attach-session -t user" in my .profile so when I login I see all the windows and can interact with them. It's important that the user processes also start with the daemon even if no user is present.

I don't understand why tmux seems to think sessions are nested just b/c 2 are started from the same script. When the script exits something is wrong and systemd will then call the script again to restart everything. For testing I do have the #Restart=on-failure commented out.

I can see the send-keys are being executed by looking at ps, they're all running. I think the TMUX & TERM env vars are key to the problem but I'm not sure how to resolve it so A) tmux separates the sessions and B) there are no problems starting up with no users or open terminal ttys.

Answer 1

This was quite the rabbit hole and took a lot of digging to figure out. IMO the systemd / OS implementors have a few items to resolve. I am not sure about Ubuntu, but Debian 8.8 definitely needs some attention.

With the info I'll provide here it shouldn't be difficult to get a working setup. Note however I do NOT have a specific answer to why tmux becomes confused and considers the above a "nested" session.

This problem arose while trying to automate the startup of a daemon process and tmux script at boot time. I attempted to use a single script to do it all and then this tmux "multiple session" problem came up.

My ultimate solution was to separate the startup of the daemon and tmux windows into 2 separate processes, both automated with systemd. I chose to use a "system" systemd unit to start the daemon and a "session" unit to start the tmux script. Although I haven't tried putting both the daemon and tmux scripts in user session systemd units, I believe that could be done without issue. There are advantages to the separation I won't go into here.

Figuring out how to utilize systemd user session units on a non-X-windows, tty only server took most of the effort to figure out. Here are the essential steps:

1) As root run apt-get install libpam-systemd. On some of the Debian 8.8 systems I've used this was necessary even after doing an apt-get update. This establishes a DBus session scope and systemd user process manager when users login, including important environment variables. Unfortunately not ALL important vars are setup and this is what the Debian OS needs to fix.

2) If you want your user session to be started at boot time so that a user does not need to login, and for it to persist across reboots, as root execute loginctl enable-linger <user account name>. This enables the [email protected] systemd process to remain even after all login sessions are closed, and will automatically start at system boot.

3) Unfortunately the important environment vars are not set correctly in Debian 8.8 and need to be fixed, however it's not difficult to do. In the user's .profile or .bashrc script, add: export DBUS_SESSION_BUS_ADDRESS="unix:path=${XDG_RUNTIME_DIR}/systemd" You should be able to inspect /run/user/$UID/ for the trailing folder that contains the sockets for user session DBus communication. On Debian 8.8 that is "systemd".

Note that you may need to reboot for all of the systemd units to initialize properly in sequence. If you execute export with no args you should see XDG_RUNTIME_DIR="/run/user/id -u" and XDG_SESSION_ID set to some value. If you don't there is still an issue with your systemd user session. If you login via ssh check that your /etc/ssh/sshd_config has UsePAM set to yes so that libpam-systemd code is executed.

I suspect the systemd units associated with libpam-systemd need to be updated so they set all of the env vars currently missing. It may be possible to make the fix in /etc/systemd/user.conf by uncommenting DefaultEnvironment= and setting it to DBUS_SESSION_BUS_ADDRESS="unix:path=${XDG_RUNTIME_DIR}/systemd", but I'm not sure that will work.

Here are a few of links you may helpful: 1: 7.10. Systemd Usage and Configuration 2: Introspecting D-Bus from the command-line 3: Manage your session with systemd