68

I would like to get a log of all processes that are launched with the time that they were launched and the arguments they were launched with. Is this possible in Linux?

0

8 Answers 8

53

Your starting point should be auditd.

Try something like this:

apt-get install auditd
auditctl -a task,always
ausearch -i -sc execve
7
  • 1
    I'm getting error The audit system is disabled Where can I enable it?
    – Tombart
    Jul 9, 2014 at 20:39
  • 1
    well one problem could be solved by chmod 0750 /sbin/audispd but it's still not working (Debian Wheezy)
    – Tombart
    Jul 9, 2014 at 20:43
  • is says Unable to set audit pid, exiting but I guess that the real problem will be that the system is running in LXC container
    – Tombart
    Jul 9, 2014 at 20:49
  • 3
    I tried this on a live server and I effectively killed it, became almost unresponsive. I barely managed to remove this rule and make the server responsive again Feb 22, 2019 at 18:37
  • 1
    @Shocker I tested it with a sleep 10 command and it generated 276 lines of output. When I consider this and your comment I certainly won't try this in production, and I need something to find out what's killing an important process.
    – Rafał G.
    Feb 11, 2020 at 7:28
14

I needed to do this, except (1) I didn't need the time and (2) I was only interested in processes that are started by a given process, and its children and subsequent descendants. Also, in the environment I was using, it wasn't possible to get auditd or accton, but there was valgrind.

Prefix the following to the process of interest on the command line:

valgrind --trace-children=yes

The information you need will be in the log output displayed on STDERR.

1
  • 7
    By default, valgrind runs with the memcheck tool. To disable the tool and its related logging, and only print the creation of new commands (in addition to the usual output of your program), use the following command instead: valgrind --tool=none --trace-children=yes [command and args here] . Whenever a subprocess is spawned, Valgrind will then log the full command, including the arguments that were passed to it.
    – Rob W
    Mar 2, 2018 at 9:39
9

You could use snoopy for this.

It is very simple to install, and since 2.x it can log arbitrary data (arguments, environmental variables, cwd, etc.).

Disclosure: Snoopy maintainer here.

0
3

You could run startmon and follow its standard output, Ctrl-C when done. Here's how to compile and run startmon on recent Red Hat derived distros (RHEL, Fedora, CentOS):

sudo yum install git cmake gcc-c++
git clone https://github.com/pturmel/startmon
cd startmon
cmake .
make
sudo ./startmon -e

On Debian (and Ubuntu etc), the first line of the above changes to:

sudo apt-get install git cmake g++

Alternatively you can try the execsnoop script in perf-tools, see this answer. By default only the first 8 arguments are shown (9 including the program name); you can increase this via

sudo ./execsnoop -a 16

If you don't have root access to the system, the best you can do is keep polling /proc and hope it catches everything (which it won't), but for completeness here's a script to do that (I've put duplicate-removal in to simplify the output)—although this isn't as good as tracking them properly with one of the above methods, it does have the slight advantage of unambiguously displaying separators between the command-line arguments, in case you ever need to tell the difference between spaces inside an argument and space between arguments. This script is inefficient as it uses the CPU (well, one of its cores) 100% of the time.

function pstail () { python -c 'import os
last=set(os.listdir("/proc")) ; o=x=""
while True:
 pids=set(os.listdir("/proc"))
 new=pids.difference(last);last=pids
 for n in new:
  try: o,x=x,[j for j in open("/proc/"+n+"/cmdline")
    .read().split(chr(0)) if j]
  except IOError: pass
  if x and not o==x: print n,x' ; }

pstail

You can also patch execsnoop to tell you more explicitly which argument is which: grep -v sub.*arg < execsnoop > n && chmod +x n && mv n execsnoop

3

CONFIG_FTRACE and CONFIG_KPROBES through brendangregg/perf-tools

git clone https://github.com/brendangregg/perf-tools.git
cd perf-tools
git checkout 98d42a2a1493d2d1c651a5c396e015d4f082eb20
sudo ./execsnoop

On another shell:

while true; do sleep 1; date; done

First shell shows data of format:

Tracing exec()s. Ctrl-C to end.                                                        
Instrumenting sys_execve                                                               
   PID   PPID ARGS 
 20109   4336 date                                                                                       
 20110   4336 sleep 1                                                                                    
 20111   4336 date                                                                                                                                                                                                 
 20112   4336 sleep 1                                                                                    
 20113   4336 date                                                                                       
 20114   4336 sleep 1                                                                                    
 20115   4336 date                                                                                       
 20116   4336 sleep 1

CONFIG_PROC_EVENTS

Sample session:

$ su
# ./proc_events &
# /proc_events.out &
set mcast listen ok
# sleep 2 & sleep 1 &
fork: parent tid=48 pid=48 -> child tid=56 pid=56
fork: parent tid=48 pid=48 -> child tid=57 pid=57
exec: tid=57 pid=57
exec: tid=56 pid=56
exit: tid=57 pid=57 exit_code=0
exit: tid=56 pid=56 exit_code=0

CONFIG_PROC_EVENTS exposes the events to userland via a netlink socket.

proc_events.c adapted from: https://bewareofgeek.livejournal.com/2945.html

#define _XOPEN_SOURCE 700
#include <sys/socket.h>
#include <linux/netlink.h>
#include <linux/connector.h>
#include <linux/cn_proc.h>
#include <signal.h>
#include <errno.h>
#include <stdbool.h>
#include <unistd.h>
#include <string.h>
#include <stdlib.h>
#include <stdio.h>

static volatile bool need_exit = false;

static int nl_connect()
{
    int rc;
    int nl_sock;
    struct sockaddr_nl sa_nl;

    nl_sock = socket(PF_NETLINK, SOCK_DGRAM, NETLINK_CONNECTOR);
    if (nl_sock == -1) {
        perror("socket");
        return -1;
    }
    sa_nl.nl_family = AF_NETLINK;
    sa_nl.nl_groups = CN_IDX_PROC;
    sa_nl.nl_pid = getpid();
    rc = bind(nl_sock, (struct sockaddr *)&sa_nl, sizeof(sa_nl));
    if (rc == -1) {
        perror("bind");
        close(nl_sock);
        return -1;
    }
    return nl_sock;
}

static int set_proc_ev_listen(int nl_sock, bool enable)
{
    int rc;
    struct __attribute__ ((aligned(NLMSG_ALIGNTO))) {
        struct nlmsghdr nl_hdr;
        struct __attribute__ ((__packed__)) {
            struct cn_msg cn_msg;
            enum proc_cn_mcast_op cn_mcast;
        };
    } nlcn_msg;

    memset(&nlcn_msg, 0, sizeof(nlcn_msg));
    nlcn_msg.nl_hdr.nlmsg_len = sizeof(nlcn_msg);
    nlcn_msg.nl_hdr.nlmsg_pid = getpid();
    nlcn_msg.nl_hdr.nlmsg_type = NLMSG_DONE;

    nlcn_msg.cn_msg.id.idx = CN_IDX_PROC;
    nlcn_msg.cn_msg.id.val = CN_VAL_PROC;
    nlcn_msg.cn_msg.len = sizeof(enum proc_cn_mcast_op);

    nlcn_msg.cn_mcast = enable ? PROC_CN_MCAST_LISTEN : PROC_CN_MCAST_IGNORE;

    rc = send(nl_sock, &nlcn_msg, sizeof(nlcn_msg), 0);
    if (rc == -1) {
        perror("netlink send");
        return -1;
    }

    return 0;
}

static int handle_proc_ev(int nl_sock)
{
    int rc;
    struct __attribute__ ((aligned(NLMSG_ALIGNTO))) {
        struct nlmsghdr nl_hdr;
        struct __attribute__ ((__packed__)) {
            struct cn_msg cn_msg;
            struct proc_event proc_ev;
        };
    } nlcn_msg;
    while (!need_exit) {
        rc = recv(nl_sock, &nlcn_msg, sizeof(nlcn_msg), 0);
        if (rc == 0) {
            /* shutdown? */
            return 0;
        } else if (rc == -1) {
            if (errno == EINTR) continue;
            perror("netlink recv");
            return -1;
        }
        switch (nlcn_msg.proc_ev.what) {
            case PROC_EVENT_NONE:
                printf("set mcast listen ok\n");
                break;
            case PROC_EVENT_FORK:
                printf("fork: parent tid=%d pid=%d -> child tid=%d pid=%d\n",
                        nlcn_msg.proc_ev.event_data.fork.parent_pid,
                        nlcn_msg.proc_ev.event_data.fork.parent_tgid,
                        nlcn_msg.proc_ev.event_data.fork.child_pid,
                        nlcn_msg.proc_ev.event_data.fork.child_tgid);
                break;
            case PROC_EVENT_EXEC:
                printf("exec: tid=%d pid=%d\n",
                        nlcn_msg.proc_ev.event_data.exec.process_pid,
                        nlcn_msg.proc_ev.event_data.exec.process_tgid);
                break;
            case PROC_EVENT_UID:
                printf("uid change: tid=%d pid=%d from %d to %d\n",
                        nlcn_msg.proc_ev.event_data.id.process_pid,
                        nlcn_msg.proc_ev.event_data.id.process_tgid,
                        nlcn_msg.proc_ev.event_data.id.r.ruid,
                        nlcn_msg.proc_ev.event_data.id.e.euid);
                break;
            case PROC_EVENT_GID:
                printf("gid change: tid=%d pid=%d from %d to %d\n",
                        nlcn_msg.proc_ev.event_data.id.process_pid,
                        nlcn_msg.proc_ev.event_data.id.process_tgid,
                        nlcn_msg.proc_ev.event_data.id.r.rgid,
                        nlcn_msg.proc_ev.event_data.id.e.egid);
                break;
            case PROC_EVENT_EXIT:
                printf("exit: tid=%d pid=%d exit_code=%d\n",
                        nlcn_msg.proc_ev.event_data.exit.process_pid,
                        nlcn_msg.proc_ev.event_data.exit.process_tgid,
                        nlcn_msg.proc_ev.event_data.exit.exit_code);
                break;
            default:
                printf("unhandled proc event\n");
                break;
        }
    }

    return 0;
}

static void on_sigint(__attribute__ ((unused)) int unused)
{
    need_exit = true;
}

int main()
{
    int nl_sock;
    int rc = EXIT_SUCCESS;

    signal(SIGINT, &on_sigint);
    siginterrupt(SIGINT, true);
    nl_sock = nl_connect();
    if (nl_sock == -1)
        exit(EXIT_FAILURE);
    rc = set_proc_ev_listen(nl_sock, true);
    if (rc == -1) {
        rc = EXIT_FAILURE;
        goto out;
    }
    rc = handle_proc_ev(nl_sock);
    if (rc == -1) {
        rc = EXIT_FAILURE;
        goto out;
    }
    set_proc_ev_listen(nl_sock, false);
out:
    close(nl_sock);
    exit(rc);
}

GitHub upsatream.

I don't think however that you can you get process data such as UID and process arguments because exec_proc_event contains so little data: https://github.com/torvalds/linux/blob/v4.16/include/uapi/linux/cn_proc.h#L80 We could try to immediately read it from /proc, but there is a risk that the process finished and another one took its PID, so it wouldn't be reliable.

Tested in Ubuntu 17.10.

1

Building on Mikel's answer here. Someone mentioned auditd ground their live server almost to a halt. To prevent this I will just run it for a short window then clear all auditing..

Install + start auditd

For RHEL:

yum install audit
systemctl start auditd

For Ubuntu/Debian:

apt install auditd
systemctl start auditd

Run audit

Run the process audit for 10 seconds, then stop the auditing. This will clear ALL rules, including any you may have defined elsewhere.

# first lets check if there are any rules in auditd
auditctl -l
# no rules? if none then we can run the following:
auditctl -a task,always; sleep 10; auditctl -D

Query the audit log

ausearch -i -sc execve > mylog.txt
0

Also you can use atop for viewing resource usage by processes. This is useful tools for logging and analyzing resource usage in each part of time

-6

You can try cat ~/.bash_history There is system log viewer,this can help you out.

2
  • 1
    ~/.bash_history only contains commands I've executed in a terminal, apparently. I'm looking for a log of all the programs executed, for example when I click an icon to open up my e-mail client, gedit, or I open up my browser, and my browser executes another process by itself. new123456's answer did the trick.
    – runeks
    Jun 1, 2011 at 19:22
  • 2
    Let's also add that the command history is the usual way of accessing this information.
    – bryn
    Apr 28, 2016 at 9:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .