Force output buffer flush in running program

Question

I have a long-running python script that periodically outputs data to standard output that I've invoked with something like:

python script.py > output.txt

This script has been running for a while and I want to stop it with Ctrl+C but not lose any of its output. Unfortunately when I implemented the script I forgot to flush the buffer after each line of output with something like the sys.stdout.flush() (the previously suggested solution for forcing output flushing), so invoking Ctrl+C right now will cause me to lose all my output.

If wondering if there's any way to interact with a running python script (or, more generally, a running process) to force it to flush its output buffer. I'm not asking how to edit and re-run the script to get it to flush correctly -- this question is specifically about interacting with a running process (and, in my case, not losing the output from my current code execution).

Answer 1

IF one were truly wanting that data, I'd suggest attaching the gdb debugger to the python interpreter, momentarily stopping the task, calling fsync(1) (stdout), detach from it (resuming the process) and go peruse the output file.

Look in /proc/$(pidof python)/fd to see valid file descriptors. $(pidof x) returns the PID of process named 'x'.

# your python script is running merrily over there.... with some PID you've determined.
#
# load gdb
gdb
#
# attach to python interpreter (use the number returned by $(pidof python))
attach 1234
#
# force a sync within the program's world (1 = stdout, which is redirected in your example)
call fsync(1)
#
# the call SHOULD have returned 0x0, sync successful.   If you get 0xffffffff (-1), perhaps that wasn't stdout.  0=stdin, 1=stdout, 2=stderr
#
# remove our claws from poor python
detach
#
# we're done!
quit

I've used this method to change working dir's, tweak settings on the fly... many things. Alas, you can only call functions which are defined in the running program, fsync works nicely though.

(gdb command 'info functions' will list all of the functions available. Be careful though. You're operating LIVE on a process.)

There is also the command peekfd (found in psmisc package on Debian Jessie and others) which will allow you to see what's hiding in buffers of a process. Again, /proc/$(pidof python)/fd will show you valid file descriptors to give as arguments to peekfd.

If you don't remember -u for python, you can always prefix a command with stdbuf (in coreutils, already installed) to set stdin/stdout/stderr to unbuffered, line buffered or block buffered as desired:

stdbuf -i 0 -o 0 -e 0 python myscript.py > unbuffered.output

Of course, man pages are your friends, hey! perhaps an alias might be useful here too.

alias python='python -u'

Now your python always uses -u for all your command line endeavors!

Answer 2

First make sure you have the debugging symbols for Python (or at least glibc). On Fedora¹ you can install them with:

dnf debuginfo-install python

Then attach gdb to the running script and run the following commands:

[user@host ~]$ pidof python2
9219
[user@host ~]$ gdb python2 9219
GNU gdb (GDB) Fedora 7.7.1-13.fc20
...
0x00007fa934278780 in __read_nocancel () at ../sysdeps/unix/syscall-template.S:81
81  T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) call fflush(stdout)
$1 = 0
(gdb) call setvbuf(stdout, 0, 2, 0)
$2 = 0
(gdb) quit
A debugging session is active.

    Inferior 1 [process 9219] will be detached.

Quit anyway? (y or n) y
Detaching from program: /usr/bin/python2, process 9219

This will flush stdout and also disable buffering. The 2 from the setvbuf call is the value of _IONBF on my system. You'll need to find out what's on yours (a grep _IONBF /usr/include/stdio.h should do the trick).

Based on what I've seen in the implementation of PyFile_SetBufSize and PyFile_WriteString in CPython 2.7, it should work pretty well, but I can't make any guarantees.

---

^{1 Fedora includes a special type of RPMs called debuginfo rpms. These automatically created RPMs contain the debugging information from the program files, but moved into an external file.}

Answer 3

There is no solution to your immediate problem. If your script has already started, you cannot change the buffering mode after the fact. These are all in-memory buffers and all of that is set up when the script starts, file handles are opened, pipes are created, etc.

As a long-shot, if and only if some or all of the buffering in question is being done at the IO level on output, you could do a sync command; but this is generally unlikely in a case like this.

In the future you can use Python's -u option^* to run the script. In general, many commands have command-specific options to disable stdin/stdout buffering, and you may also have some generic success with the unbuffer command from the expect package.

A Ctrl+C would cause system-level buffers to be flushed when the program is interrupted unless the buffering is done by Python itself and it has not implemented the logic to flush its own buffers with Ctrl+C. A suspend, crash, or kill would not be so kind.

_{^*Force stdin, stdout and stderr to be totally unbuffered.}

Answer 4

Python 2.7.7 Documentation, section "Python Setup and Usage", subsection 1. Command line and environment, describes this Python argument :

-u

Force stdin, stdout and stderr to be totally unbuffered. On systems where it matters, also put stdin, stdout and stderr in binary mode.

Note that there is internal buffering in file.readlines() and File Objects (for line in sys.stdin) which is not influenced by this option. To work around this, you will want to use file.readline() inside a while 1: loop.

And also this environment variable :

PYTHONUNBUFFERED

If this is set to a non-empty string it is equivalent to specifying the -u option.

Answer 5

It seems that I was being over-cautious about losing by buffered output after running Ctrl-C; according to this post I should expect the buffer to be flushed if my program has a normal exit, which would be the case if I hit Ctrl-C. On the other hand, I would lose the buffered output if I killed the script with SIGKILL or similar.

Answer 6

I think another possible solution can be to force the process kill with core dumped and then analyze posthumously memory content.