How to remotely terminate bash shell processes but ensure history is saved?

Question

Occasionally my X11 window manager (i3) locks up, and I'm forced to switch to a Linux virtual console, or SSH into my machine, to regain control.

Usually this happens when I have numerous rxvt windows open, each of which running a bash shell containing tens if not hundreds of lines of useful command-line history collected over many days or weeks.

I'm using shopt -s histappend so each bash shell will append its history to the history file when it terminates in an orderly manner. Normally this works very well and it will retain all of my history provided I close each shell cleanly with exit or CTRL-D.

But when my window manager locks up, I can't cleanly exit my bash sessions because there's no way to interact with them. So I've been trying to find a way to _remotely_ terminate a bash shell in a way that will cause it to write its history to the history file.

I've tried sending numerous signals, including SIGHUP, to both the bash shell process and the containing rxvt process, but none result in a history save.

I've tried to restart i3 but for various reasons this doesn't work (broken RPC socket it appears). Killing the window manager closes all the windows and bash shells and I lose the history from all of them.

I've tried to find a way to inject "CTRL-D" into each shell's stdin, but as far as I can tell that's not possible.

I know of ways to make bash save history to the history file after every command, but I prefer not to use these as they mess up the history indices (i.e. !nnn doesn't work because nnn changes after every command).

Is there a way to do this and rescue my precious history from certain oblivion?

Answer 1

Use reptyr.

reptyr is a utility for taking an existing running program and attaching it to a new terminal.

See this answer of mine for details. After you attach the desired shell process to your current terminal, you can tell it to exit gracefully.

It may not be easy to identify the particular shell process you'd like to attach at the moment though. You will probably use reptyr on any shell process found by ps (or similar tool), then you will check previous commands and only then you will know whether it's the desired session or not.

A way more convenient solution is to use tmux or screen in the first place.

I work with tmux on a daily basis and it's a trivial task to reattach from tty2. With the tool I don't need "numerous rxvt windows open". Usually I organize my work like this:

1. Single konsole window.
2. One tab per machine (local host directly, plus I ssh to few remote ones), tmux on each.
3. Inside each tmux one window or more (as needed).
4. Inside each window one pane or more (as needed).

These "layers" are marked with corresponding red numbers in the screenshot below. X11 or KDE problems don't affect my shell sessions, as long as I can get to any console of the given machine. Only a meltdown of the tmux server itself can hurt but I can't remember the last time such a thing happened to me (probably never).

In case "I'm forced to switch to a Linux virtual console, or SSH into my machine, to regain control", all I need is tmux a and I have my shell session(s) back, fully functional, as if nothing happened. If I can force my window manager to reload then I don't even need to terminate these shell processes, because I can get back to them from a new konsole, rxvt or any terminal emulator.

Answer 2

reptyr is an amazing solution if available, which is unfortunately not always the case.

When you send a HUP to the lost bash after reconnecting, it does write out its history. What typically happens though, is that you are doing the HUP sending from another bash session, which will overwrite the killed shell's history upon exit.

You want to flush your current shell's history, HUP the lost shell, give it a second to flush its own history and then reload the history from disk:

$ history -a; kill -HUP 40705; sleep 1; history -r

You will end up with full history, both local and dead shell's, in that order.

Tested on GNU bash 4.1.2(1)-release.