4

strace allows you to monitor the activity of a particular program. How can I monitor the activity of all programs (expect for sshd/bash/strace_itself)?

Currently I'm using a hack like this:

function ppid() { cat /proc/"$1"/status | grep PPid: | grep -o "[0-9]*"; };
function pidtree() { P=$1; while [ "$P" != "1" ]; do echo $P; P=`ppid $P`; done; echo 1; };
strace -fe execve `( pgrep ''; pidtree $$ ) | sort | uniq -u | sed 's/^/-p /'`

Note: using this can freeze your system.

Is there a better way to do it?

Improve this question

2 Answers 2

Reset to default
3

Simplest way with systemtap is something like:

stap -e 'probe nd_syscall.* { println(execname(), pid(), " ", pn(), argstr) }'

(@Vi, no manual kernel module work is required; systemtap does that for itself. You need kernel-module-development files available though.)

Improve this answer
5
  • The format of stap is very different from strace. "-e execve" is just for example. I may need timestamping, opened files, sockets and other things.
    – Vi.
    Jul 20, 2012 at 16:38
  • Tried this: semantic error: unresolved arity-0 function: identifier 'pn' at <input>:1:54
    – Vi.
    Jul 20, 2012 at 16:39
  • Vi., try pp() instead of pn(), if your copy of stap is older than release 1.3 (two years ago). You can control the output format with printf("..."), and add timestamping etc. I believe someone did write an approximate strace-format stap script.
    – fche
    Jul 23, 2012 at 17:22
  • Now it shows "error: `CLONE_STOPPED' undeclared here" and other C errors. It is in general more fragile way and I don't expect it to work out of the box on varied systems.
    – Vi.
    Jul 24, 2012 at 15:53
  • (Vi., please consider sending a problem report to our mailing list, <[email protected]>.)
    – fche
    Jul 27, 2012 at 19:47
1

It's not very feasible to "strace the whole system" from userspace. As I indicated in the previous question you asked, the best way is to use a kernel-mode tracing infrastructure such as kprobes, systemtap, or dtrace. Have you looked at any of these? Is there a reason why none of them will work for your use case?

The only way to truly reliably strace the entire system from userspace would be to start your trace with the init process... but I'm not sure that init or systemd would be very happy with you stracing it, since it does a lot of very low-level stuff that's pretty fragile and easy to break (and hard to inject wrapper commands around it too, I might add).

This is why the highest quality probing mechanisms have some type of kernel module, because the kernel "sees all". This is especially relevant since you are trying to monitor activity on character devices such as /dev/console and /dev/tty*, and the kernel has direct oversight over the calls to those devices since they are implemented in kernelspace.

Improve this answer
3
  • Kernel module approach is not convenient when you want to just SSH into some remote system and start monitoring.
    – Vi.
    Jul 20, 2012 at 15:40
  • Well if you're SSHing into a remote system with the intention of viewing the system calls or libc calls of all processes on the system, I have to assume that you have root access, right? Because without root you may as well give up now, because there's no way (outside of exploiting a security vulnerability) to trace processes that your user doesn't own if you aren't root. And root can load kernel modules, so.........
    – allquixotic
    Jul 20, 2012 at 15:46
  • Yes, as root, to determine what process tries to open some file, for example. The solution does not need to be suitable for production servers, just as development or personal usage tool.
    – Vi.
    Aug 6, 2012 at 1:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .