1

Something is constantly causing mischief on my system, but I have not been able to identify which program it is. It happens a few times every hour. I've found the PIDs in a log, but the program is very short lived so by the time I can ps -fp pid, it's gone.

Can I set up logging somehow to see what all PIDs were afterwards? Which program say pid 12345 was with a timestamp? I've tried running find /proc/ -maxdepth 2 -name cmdline -mmin -3 -not -type d|while read l;do echo -n $l;cat $l;done and variants of that frequently, but without success. I'm running Ubuntu 16.04.

1
  • 1
    Sounds like a job for the audit system. – Daniel B Jul 10 '20 at 10:15
1

I used to administer a public shell host where users ran all sorts of processes that occasionally went rogue. We had root cron jobs that looked like this:

 * * * * * (date; ps auxwwww) >> /var/log/ps.log
 * * * * * (date; lsof -n) >> /var/log/lsof.log

We also installed rotation for the two log files so they wouldn't overflow.

When we observed problems with the system, we would review the ps log to catch the name of the rogue process and its arguments, and the lsof logs to find the process's executable image and any interesting files it had open.

However, if the rogue process you're looking for runs for a few seconds or less, then it's not likely to get caught by the above scripts which only run once a minute. You can write your own monitoring script that runs the above commands at a faster rate, say once every second.

If that's still not enough to catch it, you'll need something that vacuums up data from every process in the system. An Ubuntu 16.04 machine should be able to run relatively recent performance tools like bpf:

http://www.brendangregg.com/blog/2016-06-14/ubuntu-xenial-bcc-bpf.html

If that's not an option, you could try to use strace on the rogue process's parent process, assuming you knew what it was. If it's something being run out of cron or a user logging via ssh, a carefully written strace command that only captures process events and ignores everything else might work - but be very careful as it could be disruptive, especially if you trace too many things at once.

Finally, you don't specify what "mischief" is happening on your system - there are ways to trigger commands based on system events (update of a file, appearance of a packet on the network interface, CPU getting too high) that might work for you depending on your symptoms.

2
  • Thx for commenting, but once a minute is way too infrequent. The process takes less than 0.1 sec to finish. I found the very guilty program (which used a wrong APIkey and got me suspended from that service for an hour at a time). But to learn more about Linux I would still be interested to know if there is a way to log every new PID that is started with program path and timestamp. – Kjetil S. Feb 11 '19 at 16:05
  • You can get almost what you want with Linux process accounting - see tecmint.com/… It captures the user and program name, but not the PID. – Velo Traveler Mar 4 '19 at 2:25
0

You can try to make a cron task that extract new process and log it to a file.

For example a command executing every minutes doing a ps wich select only process younger than the last cron execution and log it in a file. With chance you will catch it.

2
  • I think that is a variant of the find command I described. I'll guess I'll just let it run for a longer period and eventually I'll get lucky. – Kjetil S. Feb 9 '19 at 17:53
  • The point is to try the command many time to cath it one time, one or orher command is not important – redheness Feb 9 '19 at 18:03
0

First I run this to get an idea of how many PIDs are being consumed in a 10 second period. You can adjust the sleep to your needs.

$ bash -c 'echo $$'; sleep 10; bash -c 'echo $$'

Sample output. This tells me there were almost 200 PIDs consumed in 10 seconds, about 20 PID/sec.

11355
11528

Here's what I use to find what is consuming the PIDs. I use the "L" option to include threads. Again, you can adjust the sleep.

$ ps -eLf | awk '{print $1,$2,$3,$8}' > x1; sleep 0.5; ps -eLf | awk '{print $1,$2,$3,$8}' > x2; diff x[12]

Here's what the output looks like when I don't catch anything besides the commands I'm running.

1149,1150c1149,1150
< mudd 18109 1684 ps
< mudd 18110 1684 awk
---
> mudd 18187 1684 ps
> mudd 18188 1684 awk

That's my quick & dirty approach. Here's a Python script that does the same thing in an endless loop. The psutil module has to be added to your Python2 using pip install psutil for this to work. I added a time.sleep at the end of the loop to slow it down but that's optional.

#! /usr/bin/env python

import datetime
import time

import psutil

prev_pid2details = {}
pid2details = {}
first_pass = True

while True:
    for p in psutil.process_iter():
        try:
            start_time = str(datetime.datetime.fromtimestamp(p.create_time()))[:22]
            key = (p.pid, start_time)
            if not first_pass and key not in prev_pid2details:
                print key, p.ppid(), p.name(), p.cmdline()
            pid2details[key] = (p.ppid(), p.name(), p.cmdline())
        except psutil.NoSuchProcess:
            continue
    print '---'
    prev_pid2details = pid2details
    pid2details = {}
    first_pass = False

    time.sleep(0.3)

The above approach is not foolproof. Here's an example of a way to consume PIDs that can't be traced by comparing output from ps command or psutil.

$ while true; do bash -c 'exit'; done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.