99 % CPU taken up by “ps”

Question

I have an EC2 t2.micro (Linux AMI) instance. Since a few days I noticed that RAM is being 99% used by "ps" .

When that happens :

$ free -m
              total        used        free      shared  buff/cache   available
Mem:            983         561          79           0         342         257
Swap:             0           0           0

top shows

$ top
top - 05:58:52 up 17:28,  2 users,  load average: 1.00, 0.78, 0.39
Tasks:  95 total,   1 running,  58 sleeping,   0 stopped,   0 zombie
%Cpu(s):100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1006940 total,    80816 free,   575132 used,   350992 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   263376 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
14386 ec2-user  20   0  307096 269360   4192 S 99.6 26.8   7:15.88 ps
 9019 ec2-user  20   0  771132 128884  47792 S  0.4 12.8   1:18.75 python
    1 root      20   0   45624   5372   3936 S  0.0  0.5   0:07.01 systemd
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.05 kthreadd
    4 root       0 -20       0      0      0 I  0.0  0.0   0:00.00 kworker/0:0H
    6 root       0 -20       0      0      0 I  0.0  0.0   0:00.00 mm_percpu_wq
    7 root      20   0       0      0      0 S  0.0  0.0   0:00.52 ksoftirqd/0
    8 root      20   0       0      0      0 I  0.0  0.0   0:03.64 rcu_sched
    9 root      20   0       0      0      0 I  0.0  0.0   0:00.00 rcu_bh
   10 root      rt   0       0      0      0 S  0.0  0.0   0:00.00 migration/0
   11 root      rt   0       0      0      0 S  0.0  0.0   0:00.15 watchdog/0
   12 root      20   0       0      0      0 S  0.0  0.0   0:00.00 cpuhp/0
   14 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kdevtmpfs
   15 root       0 -20       0      0      0 I  0.0  0.0   0:00.00 netns
   16 root      20   0       0      0      0 I  0.0  0.0   0:00.33 kworker/u30:1
   21 root      20   0       0      0      0 S  0.0  0.0   0:00.00 xenbus
   22 root      20   0       0      0      0 S  0.0  0.0   0:00.06 xenwatch
  172 root      20   0       0      0      0 S  0.0  0.0   0:00.01 khungtaskd
  173 root      20   0       0      0      0 S  0.0  0.0   0:00.00 oom_reaper
  174 root       0 -20       0      0      0 I  0.0  0.0   0:00.00 writeback
  176 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kcompactd0
  177 root      25   5       0      0      0 S  0.0  0.0   0:00.00 ksmd
  178 root      39  19       0      0      0 S  0.0  0.0   0:00.00 khugepaged

Once I kill the ps process the system comes to normal state .

$ free -m
              total        used        free      shared  buff/cache   available
Mem:            983         294         343           0         345         524
Swap:             0           0           0

[ec2-user@ip-10-0-0-86 scenAI]$ top
top - 06:05:36 up 17:35,  1 user,  load average: 0.00, 0.20, 0.25
Tasks:  88 total,   1 running,  51 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1006940 total,   351520 free,   301488 used,   353932 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   537044 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 9017 ec2-user  20   0  231592  21596   9392 S  0.3  2.1   1:32.00 uvicorn
 9019 ec2-user  20   0  771132 128884  47792 S  0.3 12.8   1:19.29 python
    1 root      20   0   45624   5372   3936 S  0.0  0.5   0:07.03 systemd
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.05 kthreadd

Though I am not directly running "ps" command random calls seem to be invoking it and once the script fail, checking "top" shows "ps" running at 99% CPU .

Also previously **ps -ef|grep ** used to work but now gives ps: unrecognized option: e ps: unrecognized option: f .

ec2-user@xxx ~]$ top : PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 18096 ec2-user 20 0 307088 4932 3992 S 99.3 0.5 0:23.54 ps 

cat /proc/18096/cmdline | strings -1 

[ec2-user@xxx ~]$ cat /proc/18096/cmdline | strings -1 ERROR: ld.so: object 'al/lib/libprocesshider.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored. ERROR: ld.so: object 'al/lib/libprocesshider.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored. 
ps 
-ef 

Appreciate any pointers.

regards, Thomas

Answer 1

When you ran cat /proc/18096/cmdline | strings -1, a curious error message popped up:

ERROR: ld.so: object 'al/lib/libprocesshider.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored. ERROR: ld.so: object 'al/lib/libprocesshider.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.

The error probably appears twice because two commands are in the pipe.

While libprocesshider itself is open source software, you would not normally have it on your system. It is very likely your system is infected with malware.

libprocesshider may have malfunctioned within ps, triggering the problem your initially observed.

The safest way to proceed is to back up any application data you may not have locally and then recreate the EC2 VM from scratch.

You should also review your security. Somehow, an attacker was able to write to /etc/ld.so.preload, a file that is writable only by root. Maybe you should create a dedicated user for your application (instead of the ec2-user account) which would only have the minimum required permissions. If, for whatever reason, your application must use sudo (or some similar mechanism) to run commands as root, make sure it can only run the commands it has to with the exact parameters it needs.