18

I have a long-running python script that periodically outputs data to standard output that I've invoked with something like:

python script.py > output.txt

This script has been running for a while and I want to stop it with Ctrl+C but not lose any of its output. Unfortunately when I implemented the script I forgot to flush the buffer after each line of output with something like the sys.stdout.flush() (the previously suggested solution for forcing output flushing), so invoking Ctrl+C right now will cause me to lose all my output.

If wondering if there's any way to interact with a running python script (or, more generally, a running process) to force it to flush its output buffer. I'm not asking how to edit and re-run the script to get it to flush correctly -- this question is specifically about interacting with a running process (and, in my case, not losing the output from my current code execution).

16
+50

IF one were truly wanting that data, I'd suggest attaching the gdb debugger to the python interpreter, momentarily stopping the task, calling fsync(1) (stdout), detach from it (resuming the process) and go peruse the output file.

Look in /proc/$(pidof python)/fd to see valid file descriptors. $(pidof x) returns the PID of process named 'x'.

# your python script is running merrily over there.... with some PID you've determined.
#
# load gdb
gdb
#
# attach to python interpreter (use the number returned by $(pidof python))
attach 1234
#
# force a sync within the program's world (1 = stdout, which is redirected in your example)
call fsync(1)
#
# the call SHOULD have returned 0x0, sync successful.   If you get 0xffffffff (-1), perhaps that wasn't stdout.  0=stdin, 1=stdout, 2=stderr
#
# remove our claws from poor python
detach
#
# we're done!
quit

I've used this method to change working dir's, tweak settings on the fly... many things. Alas, you can only call functions which are defined in the running program, fsync works nicely though.

(gdb command 'info functions' will list all of the functions available. Be careful though. You're operating LIVE on a process.)

There is also the command peekfd (found in psmisc package on Debian Jessie and others) which will allow you to see what's hiding in buffers of a process. Again, /proc/$(pidof python)/fd will show you valid file descriptors to give as arguments to peekfd.

If you don't remember -u for python, you can always prefix a command with stdbuf (in coreutils, already installed) to set stdin/stdout/stderr to unbuffered, line buffered or block buffered as desired:

stdbuf -i 0 -o 0 -e 0 python myscript.py > unbuffered.output

Of course, man pages are your friends, hey! perhaps an alias might be useful here too.

alias python='python -u'

Now your python always uses -u for all your command line endeavors!

5

First make sure you have the debugging symbols for Python (or at least glibc). On Fedora1 you can install them with:

dnf debuginfo-install python

Then attach gdb to the running script and run the following commands:

[user@host ~]$ pidof python2
9219
[user@host ~]$ gdb python2 9219
GNU gdb (GDB) Fedora 7.7.1-13.fc20
...
0x00007fa934278780 in __read_nocancel () at ../sysdeps/unix/syscall-template.S:81
81  T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS)
(gdb) call fflush(stdout)
$1 = 0
(gdb) call setvbuf(stdout, 0, 2, 0)
$2 = 0
(gdb) quit
A debugging session is active.

    Inferior 1 [process 9219] will be detached.

Quit anyway? (y or n) y
Detaching from program: /usr/bin/python2, process 9219

This will flush stdout and also disable buffering. The 2 from the setvbuf call is the value of _IONBF on my system. You'll need to find out what's on yours (a grep _IONBF /usr/include/stdio.h should do the trick).

Based on what I've seen in the implementation of PyFile_SetBufSize and PyFile_WriteString in CPython 2.7, it should work pretty well, but I can't make any guarantees.


1 Fedora includes a special type of RPMs called debuginfo rpms. These automatically created RPMs contain the debugging information from the program files, but moved into an external file.

  • I tried python 2.7 and ended up with the same result. I'll take a look at the debugging update you posted. – DarkHeart Dec 2 '16 at 4:32
  • For what it's worth, CPython 3.5 seems to have a different implementation of I/O (fileobject.c) than 2.7. Someone needs to dig into the io module. – Cristian Ciupitu Dec 2 '16 at 4:50
  • @DarkHeart, you might want to test first with a simple program like this one. – Cristian Ciupitu Dec 2 '16 at 5:01
4

There is no solution to your immediate problem. If your script has already started, you cannot change the buffering mode after the fact. These are all in-memory buffers and all of that is set up when the script starts, file handles are opened, pipes are created, etc.

As a long-shot, if and only if some or all of the buffering in question is being done at the IO level on output, you could do a sync command; but this is generally unlikely in a case like this.

In the future you can use Python's -u option* to run the script. In general, many commands have command-specific options to disable stdin/stdout buffering, and you may also have some generic success with the unbuffer command from the expect package.

A Ctrl+C would cause system-level buffers to be flushed when the program is interrupted unless the buffering is done by Python itself and it has not implemented the logic to flush its own buffers with Ctrl+C. A suspend, crash, or kill would not be so kind.

*Force stdin, stdout and stderr to be totally unbuffered.

2

Python 2.7.7 Documentation, section "Python Setup and Usage", subsection 1. Command line and environment, describes this Python argument :

-u

Force stdin, stdout and stderr to be totally unbuffered. On systems where it matters, also put stdin, stdout and stderr in binary mode.

Note that there is internal buffering in file.readlines() and File Objects (for line in sys.stdin) which is not influenced by this option. To work around this, you will want to use file.readline() inside a while 1: loop.

And also this environment variable :

PYTHONUNBUFFERED

If this is set to a non-empty string it is equivalent to specifying the -u option.

  • 1
    Thanks -- but these both sound like options that I would need to specify when I first ran my python script. I'm wondering if there's a way to get a running script to dump its output. – josliber Jun 13 '14 at 17:22
  • I don't believe there is such a solution, because the data is probably in a memory buffer somewhere. You would need to inject a dll into python that knows its executable well enough to know where is the buffer and how to write it out. I believe most people would just use one of the above 2 methods. Adding an environment variable is rather easy, after all. – harrymc Jun 13 '14 at 19:15
  • OK, good to know there may not be a solution. As stated in my question, I know how to flush buffers in python (I would have used sys.stdout.flush(), but your -u option seems even easier), but had just forgotten to do so when invoking my code. Having already run my code for more than a week, I was hoping there was a way to get my output without needing to re-run the code for another week. – josliber Jun 13 '14 at 19:42
  • A far-fetched method, if you know what the data looks like, is to take a full memory dump of the process using Process Explorer, then search for the strings in the file. This won't terminate the process, so you can still try other methods. – harrymc Jun 13 '14 at 21:09
  • I'm on linux -- are there linux equivalents of that software? – josliber Jun 13 '14 at 21:43
2

It seems that I was being over-cautious about losing by buffered output after running Ctrl-C; according to this post I should expect the buffer to be flushed if my program has a normal exit, which would be the case if I hit Ctrl-C. On the other hand, I would lose the buffered output if I killed the script with SIGKILL or similar.

  • You'd have to try it to find out. Ctrl-C will cause low-level IO buffers to be flushed. If Python does its own buffering then Ctrl-C will only flush them if Python is kind enough to implement the logic to do so. Hopefully Python decided not to reinvent a wheel and relies on the system's normal level of buffering. I have no idea if that's the case. But be warned. – Jason C Jun 16 '14 at 15:47
  • The OS can never flush out what is in the program's memory space. What gets flushed is data in system memory, meaning data already written out by the program using system calls. In case of an error exit, even these system buffers are discarded. In short, data not yet written out by Python cannot be flushed and is lost in all cases. – harrymc Jun 16 '14 at 18:30
0

I think another possible solution can be to force the process kill with core dumped and then analyze posthumously memory content.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.