Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

strace allows you to monitor the activity of a particular program. How can I monitor the activity of all programs (expect for sshd/bash/strace_itself)?

Currently I'm using a hack like this:

function ppid() { cat /proc/"$1"/status | grep PPid: | grep -o "[0-9]*"; };
function pidtree() { P=$1; while [ "$P" != "1" ]; do echo $P; P=`ppid $P`; done; echo 1; };
strace -fe execve `( pgrep ''; pidtree $$ ) | sort | uniq -u | sed 's/^/-p /'`

Note: using this can freeze your system.

Is there a better way to do it?

share|improve this question
add comment

2 Answers 2

It's not very feasible to "strace the whole system" from userspace. As I indicated in the previous question you asked, the best way is to use a kernel-mode tracing infrastructure such as kprobes, systemtap, or dtrace. Have you looked at any of these? Is there a reason why none of them will work for your use case?

The only way to truly reliably strace the entire system from userspace would be to start your trace with the init process... but I'm not sure that init or systemd would be very happy with you stracing it, since it does a lot of very low-level stuff that's pretty fragile and easy to break (and hard to inject wrapper commands around it too, I might add).

This is why the highest quality probing mechanisms have some type of kernel module, because the kernel "sees all". This is especially relevant since you are trying to monitor activity on character devices such as /dev/console and /dev/tty*, and the kernel has direct oversight over the calls to those devices since they are implemented in kernelspace.

share|improve this answer
    
Kernel module approach is not convenient when you want to just SSH into some remote system and start monitoring. –  Vi. Jul 20 '12 at 15:40
    
Well if you're SSHing into a remote system with the intention of viewing the system calls or libc calls of all processes on the system, I have to assume that you have root access, right? Because without root you may as well give up now, because there's no way (outside of exploiting a security vulnerability) to trace processes that your user doesn't own if you aren't root. And root can load kernel modules, so......... –  ÃŁŁǫǛȉЖΦΤїҪ Jul 20 '12 at 15:46
    
Yes, as root, to determine what process tries to open some file, for example. The solution does not need to be suitable for production servers, just as development or personal usage tool. –  Vi. Aug 6 '12 at 1:29
add comment

Simplest way with systemtap is something like:

stap -e 'probe nd_syscall.* { println(execname(), pid(), " ", pn(), argstr) }'

(@Vi, no manual kernel module work is required; systemtap does that for itself. You need kernel-module-development files available though.)

share|improve this answer
    
The format of stap is very different from strace. "-e execve" is just for example. I may need timestamping, opened files, sockets and other things. –  Vi. Jul 20 '12 at 16:38
    
Tried this: semantic error: unresolved arity-0 function: identifier 'pn' at <input>:1:54 –  Vi. Jul 20 '12 at 16:39
    
Vi., try pp() instead of pn(), if your copy of stap is older than release 1.3 (two years ago). You can control the output format with printf("..."), and add timestamping etc. I believe someone did write an approximate strace-format stap script. –  fche Jul 23 '12 at 17:22
    
Now it shows "error: `CLONE_STOPPED' undeclared here" and other C errors. It is in general more fragile way and I don't expect it to work out of the box on varied systems. –  Vi. Jul 24 '12 at 15:53
    
(Vi., please consider sending a problem report to our mailing list, <systemtap@sourceware.org>.) –  fche Jul 27 '12 at 19:47
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.