0

I'm running my java program on a 12 cores 24 thread machine. They have several processes that are running simultaneously. It seems that I performed too many processes so that the whole tasks made the machine very slow.

Here is the top information

Tasks: 556 total,   2 running, 554 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.1%us,  0.4%sy,  0.0%ni, 63.2%id, 36.3%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16295248k total, 16169560k used,   125688k free,     3300k buffers
Swap: 18530296k total, 10867972k used,  7662324k free,    46188k cached

It seems that my processes is memory-consuming-oriented so that almost all the memory was used by them. In the top information what is I don't understand is why only 2 tasks are running instead of 23 (I have dispatched 23 processes).

free -g
             total       used       free     shared    buffers     cached
Mem:            15         15          0          0          0          0
-/+ buffers/cache:         15          0
Swap:           17         10          7

It seems that all the memory was used and it was swapping made the machine slow down.

ps -e -o pid,%cpu,%mem,vsz,rss,comm= --sort=vsz
29707  5.6  4.2 6268732 685660 java
29712  5.2  3.9 6268732 647352 java
...
30269  3.2  4.3 6268732 704676 java
30334  4.8  4.2 6268732 689544 java

There are 23 such java processes. Summing all the %cpu, it is very close to 100%. But the top information indicating that the CPU is not busy.

Cpu(s):  0.1%us,  0.4%sy,  0.0%ni, 63.2%id, 36.3%wa,  0.0%hi,  0.0%si,  0.0%st

I googled what size of vsz and rss but didn't find out. I assume that the unit is in kilo byte. Watching the vsz then the java processes are using 6268732kb*23=144,180,836 =~ 144gb, which seems to be impossible to put in the RAM because it is far more then my RAM (16gb), so only 700000kb*23 =~ 16gb were put into the memory (with the rss info, which is the portion of the data store in the RAM). Because of the frequent swap and context switch made the system slow down.

I don't know my conclusion is correct or not. Please give me some advice and how may I fix the problem.

Add more detail:

vmstat -a -S M
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa st
 2 29  16792    124   2105  13152    0    0    29    23    2    0  1  0 95  4  0

I don't know how should I interpret the vmstat data. It's a little bit weird because swpd: the amount of virtual memory used looks high while si and so are 0.

1
  • It looks like the swap is really being heavily used. Notice that only about 150 MB of RAM is not being used for processes! While having 10 GB on swap -> you are short on RAM. --- Use vmstat to see the actual virtual memory page faults (si and so). Use vmstat 1 to show it continuously in a second interval. vmstat also shows IO activity. Use iotop for more detailed per-process IO activity monitoring. Dec 18, 2013 at 9:01

2 Answers 2

0

You system is clearly missing RAM.

  • add more RAM (hoping you don't need to add up to 128 GB)
  • limit the number of JVM instances as each one uses 6 GB of virtual memory.
  • tune them to use less memory, these are 64 bit JVMs, have a look to the -Xmx flag.

There is no CPU issue.

6
  • Then why summing all the %cpu is very close to 100%? Dec 18, 2013 at 9:28
  • 1
    Well, I would expect the sum to be between 1200% to 2400% on a CPU bound system.
    – jlliagre
    Dec 18, 2013 at 9:36
  • You mean the % is in term of one core instead of whole system's cpu? Dec 18, 2013 at 9:46
  • In therm of (hardware) thread even.
    – jlliagre
    Dec 18, 2013 at 9:47
  • So for the 24 thread machine the maximum %cpu capacity is 2400 is ps information? Dec 18, 2013 at 10:01
0

Your conclusion is indeed correct. The memory sizes are in KB, both vsz and rss, as you can check yourself by looking at the Man page for ps, in the section STANDARD FORMAT SPECIFIERS.

There is also another check you can perform: the rss (resident set size, i.e. the non-swap memory used by each process) is about 700MB per process. If you have 23 such processes, that accounts nicely for the 15GB of used memory (not swap).

Also, the total size of your physical memory + swap is way smaller than required by the simultaneous execution of these 23 tasks, 16GB vs. 144GB. So it appears that not even a single process is allocated the required amount of memory at this time.

So what are your options? Simply run two processes at a time, since their size is such that you can keep them completely in your memory, without swapping. When they are done, load two more. This can easily be accomplished with a bash script, with the command wait:

   my_job < file1.txt &
   my_job < file2.txt &
   wait 1 2
   my_job < file3.txt
   my_job < file4.txt
   wait 3 4....

This will also will leave you some room in memory for keeping there /tmp, /run, and so on, which means the ability of your system to perform interactively will be little affected.

The second option (perhaps the first one), would be to ask yourself how did you end up with a 7GB java code... but that is a question for StackOverflow

Edit:

I am replying here to Marcus Thornton's comment:

If I can fix the memory issue, ..., is it good to dispatch all the tasks at one time?

Yes and no. If you fix the memory issue, then you may certainly execute more than 2 jobs simultaneously. But can you execute 23? I think you are overestimating the usefulness of the Multi-threading. Multi-threading is a single processor, with two execution contexts, which allows parallel use of different functional units on the same processor. But if the tasks are similar, as they are bound to be in your case, they will most likely use the same functional units, thus they will be queued just like they would be if no multi-threading existed. Multi-threading is not the same thing as having to independent processors, and it allows marginal speed gains, at best.

BTW, how may cpus do you have? You said processors, not cpu. The following command

  /bin/cat /proc/cpuinfo | /bin/egrep 'processor|model name|cache size|core|sibling|physical'

will help you.

12
  • VSZ (summed as 144 GB) is not certainly a memory required by a process! It is the allocated virtual memory space of a process which does not need to correspond to an allocated physical memory at all (RAM, swap, memory mapped files). This number is in most cases useless. Dec 18, 2013 at 9:42
  • With your solution I have to find out the pid of each jobs so that I could let the task to wait for them, right? Dec 18, 2013 at 9:44
  • @MarcusThornton: wait is a shell built-in. You can use either wait %job_n or wait pid. The easiest way would be just wait which waits for all background jobs of the shell. You will not be able to use this simple solution if you run some additional background jobs. Dec 18, 2013 at 9:53
  • 1
    @pabouk, Java doesn't allocate physical memory (RAM), it allocs (reserves, possibly with overcommitment) virtual memory just like other regular userland processes. The vsz statistic is definitely useful, especially with JVMs which do not tend to reuse the same allocated zones for new objects.
    – jlliagre
    Dec 18, 2013 at 9:56
  • @MarcusThonton Yes, but it is very easy: after job &, the shell variable $! has the PID of the job. Thus: job1 & ; PID1=$!; job2 &; PID2=$!; wait $PID1 $PID2. that's it. Dec 18, 2013 at 10:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .