How do I find out which process is causing high CPU usage?

Question

I have a strange issue happening on my Solaris virtual machine: After it is booted and up for 1-2 hours, the CPU usage will go up to 100% for 5 seconds, then down to normal for another 5 seconds, and repeats like that until a reboot - it makes my Solaris virtual machine totally unusable.

I wish to find out what is happening during the repeating 5 seconds 100% CPU usage, but the system is totally unresponsive during the 5 seconds - not even mouse/keyboard interrupts are handled, and thus I cannot see the process name using top or prstat.

So I wish to find out:

• the process id which caused the 100% CPU usage
• what the process was doing during the 100% CPU usage

Please offer your suggestions, thank you!

Answer 1

You could try running top in batch mode:

top -b -n100 > top.log

where -n100 stands for 100 iterations.

Another alternative is to use ps with appropriate arguments (these are from Linux, you may need to consult the man page of ps in Solaris). For convenience the command is embedded into a Bash script.

   #!/bin/bash 
   while true ; do
      ps -eo pcpu,pmem,pid,ppid,args >> process.log
      sleep 1
   done

You may also have to change the priority of the process activity logging process to real time with the nice command.

Answer 2

Especially as you are running in a virtualized environment, you shouldn't directly assume the culprit is a process.

This can also be an hypervisor issue or an kernel related one.

I would use dtrace to figure out what the kernel is doing during these high CPU usage periods:

The DTraceToolkit cputimes and modcalls.d commands would be a good start.

If your host OS is also Solaris, dtrace can also be useful identifying the CPU load origin