1

I have an EC2 t2.micro (Linux AMI) instance. Since a few days I noticed that RAM is being 99% used by "ps" .

When that happens :

$ free -m
              total        used        free      shared  buff/cache   available
Mem:            983         561          79           0         342         257
Swap:             0           0           0

top shows

$ top
top - 05:58:52 up 17:28,  2 users,  load average: 1.00, 0.78, 0.39
Tasks:  95 total,   1 running,  58 sleeping,   0 stopped,   0 zombie
%Cpu(s):100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1006940 total,    80816 free,   575132 used,   350992 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   263376 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
14386 ec2-user  20   0  307096 269360   4192 S 99.6 26.8   7:15.88 ps
 9019 ec2-user  20   0  771132 128884  47792 S  0.4 12.8   1:18.75 python
    1 root      20   0   45624   5372   3936 S  0.0  0.5   0:07.01 systemd
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.05 kthreadd
    4 root       0 -20       0      0      0 I  0.0  0.0   0:00.00 kworker/0:0H
    6 root       0 -20       0      0      0 I  0.0  0.0   0:00.00 mm_percpu_wq
    7 root      20   0       0      0      0 S  0.0  0.0   0:00.52 ksoftirqd/0
    8 root      20   0       0      0      0 I  0.0  0.0   0:03.64 rcu_sched
    9 root      20   0       0      0      0 I  0.0  0.0   0:00.00 rcu_bh
   10 root      rt   0       0      0      0 S  0.0  0.0   0:00.00 migration/0
   11 root      rt   0       0      0      0 S  0.0  0.0   0:00.15 watchdog/0
   12 root      20   0       0      0      0 S  0.0  0.0   0:00.00 cpuhp/0
   14 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kdevtmpfs
   15 root       0 -20       0      0      0 I  0.0  0.0   0:00.00 netns
   16 root      20   0       0      0      0 I  0.0  0.0   0:00.33 kworker/u30:1
   21 root      20   0       0      0      0 S  0.0  0.0   0:00.00 xenbus
   22 root      20   0       0      0      0 S  0.0  0.0   0:00.06 xenwatch
  172 root      20   0       0      0      0 S  0.0  0.0   0:00.01 khungtaskd
  173 root      20   0       0      0      0 S  0.0  0.0   0:00.00 oom_reaper
  174 root       0 -20       0      0      0 I  0.0  0.0   0:00.00 writeback
  176 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kcompactd0
  177 root      25   5       0      0      0 S  0.0  0.0   0:00.00 ksmd
  178 root      39  19       0      0      0 S  0.0  0.0   0:00.00 khugepaged

Once I kill the ps process the system comes to normal state .

$ free -m
              total        used        free      shared  buff/cache   available
Mem:            983         294         343           0         345         524
Swap:             0           0           0
[ec2-user@ip-10-0-0-86 scenAI]$ top
top - 06:05:36 up 17:35,  1 user,  load average: 0.00, 0.20, 0.25
Tasks:  88 total,   1 running,  51 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.0 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1006940 total,   351520 free,   301488 used,   353932 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   537044 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 9017 ec2-user  20   0  231592  21596   9392 S  0.3  2.1   1:32.00 uvicorn
 9019 ec2-user  20   0  771132 128884  47792 S  0.3 12.8   1:19.29 python
    1 root      20   0   45624   5372   3936 S  0.0  0.5   0:07.03 systemd
    2 root      20   0       0      0      0 S  0.0  0.0   0:00.05 kthreadd

Though I am not directly running "ps" command random calls seem to be invoking it and once the script fail, checking "top" shows "ps" running at 99% CPU .

Also previously **ps -ef|grep ** used to work but now gives ps: unrecognized option: e ps: unrecognized option: f .

ec2-user@xxx ~]$ top : PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 18096 ec2-user 20 0 307088 4932 3992 S 99.3 0.5 0:23.54 ps 


cat /proc/18096/cmdline | strings -1 

[ec2-user@xxx ~]$ cat /proc/18096/cmdline | strings -1 ERROR: ld.so: object 'al/lib/libprocesshider.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored. ERROR: ld.so: object 'al/lib/libprocesshider.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored. 
ps 
-ef 

Appreciate any pointers.

regards, Thomas

8
  • What’s the output of which ps? What is the exact command line when ps hangs? You can check it in /proc/<pid>/cmdline. – Daniel B Feb 23 at 9:24
  • which returns => /usr/bin/ps. – user14013917 Feb 23 at 10:33
  • [ec2-user@xxx ~]$ top : PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 18096 ec2-user 20 0 307088 4932 3992 S 99.3 0.5 0:23.54 ps ************** cat /proc/18096/cmdline | strings -1 [ec2-user@xxx ~]$ cat /proc/18096/cmdline | strings -1 ERROR: ld.so: object 'al/lib/libprocesshider.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored. ERROR: ld.so: object 'al/lib/libprocesshider.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored. ps -ef – user14013917 Feb 23 at 10:41
  • PID - 18096 CPU 99.3% . I did not find a way to edit the above o/p but the "/proc/18096/cmdline" gives ps -ef . Every command also gives error => ERROR: ld.so: object 'al/lib/libprocesshider.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored – user14013917 Feb 23 at 10:42
  • 1
    Please add additional information in your question. You can always edit it. You can leave a comment to notify watchers. // Your machine may have been compromised. – Daniel B Feb 23 at 10:54
0

When you ran cat /proc/18096/cmdline | strings -1, a curious error message popped up:

ERROR: ld.so: object 'al/lib/libprocesshider.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored. ERROR: ld.so: object 'al/lib/libprocesshider.so' from /etc/ld.so.preload cannot be preloaded (cannot open shared object file): ignored.

The error probably appears twice because two commands are in the pipe.

While libprocesshider itself is open source software, you would not normally have it on your system. It is very likely your system is infected with malware.

libprocesshider may have malfunctioned within ps, triggering the problem your initially observed.

The safest way to proceed is to back up any application data you may not have locally and then recreate the EC2 VM from scratch.

You should also review your security. Somehow, an attacker was able to write to /etc/ld.so.preload, a file that is writable only by root. Maybe you should create a dedicated user for your application (instead of the ec2-user account) which would only have the minimum required permissions. If, for whatever reason, your application must use sudo (or some similar mechanism) to run commands as root, make sure it can only run the commands it has to with the exact parameters it needs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.