11

We have embedded Linux system without swap.

Currenly we must raise alarm when memory usage % increases over a threashold. And reboot when memory usage % increases over a (higher) threshold.

Why we want to do that: If some program leaks, we can do safety reboot, before kernel start killing our processes (which may lead to data corruption or unavailability).

But we have a problem:

How to count memory usage-% which can be used for our purpose?

We tried to count memory usage by using values of /proc/meminfo:

/ # cat /proc/meminfo
MemTotal:       126744 kB
MemFree:         58256 kB
Buffers:         16740 kB
Cached:          31308 kB
SwapCached:          0 kB
Active:          37580 kB
Inactive:        24000 kB

Without success:

(MemTotal - MemFree) is not usable, because it contains for example caches.

(MemTotal - MemFree - Buffers - Cached) did ignore effect of Inactive. So it also gives too big memory usage values.

(MemTotal - MemFree - Buffers - Cached - Inactive) is unusable, because result can be negative.

4 Answers 4

6

Monitor system via free

[root@localhost ~]# free
          total       used       free     shared    buffers     cached
Mem:    2058240    1776788     281452          0      89780    1335840
-/+ buffers/cache:  351168    1707072
Swap:   4095992        100    4095892

Look at the -/+ buffers/cache line used and free

Monitor each process via /proc

I used this python script and /proc/pid/stat to monitor the memory of a process:

http://phacker.org/2009/02/20/monitoring-virtual-memory-usage-with-python/

you would probably like to translate something like this to c.

Limit resource for each process

or use ulimit / setrlimit

https://stackoverflow.com/questions/4983120/limit-memory-usage-for-a-single-linux-process

5
  • Can you explain the method your Python script uses to calculate the memory usage? That would make this a much better answer.
    – Flimzy
    May 2, 2012 at 13:33
  • Well, it just logs the vm-usage in steps of a second. I used that for graphing the mem consumption over program lifetime. This was handy to debug mem leaks in long running programms.
    – snies
    May 2, 2012 at 13:37
  • You could just use this to monitor a prog after some init time. And alert some "leak suspect flag" if vmusage crosses certain treshold.
    – snies
    May 2, 2012 at 13:39
  • 1
    The link to phacker.org is no more
    – f01
    Jul 18, 2015 at 14:28
  • ...which is why StarckExchange always asks to post the contents of scripts, not just links
    – JDS
    Apr 1, 2016 at 20:35
5
#!/bin/bash

threshold=90
threshold2=95

freemem=$(($(free -m |awk 'NR==2 {print $3}') * 100))

usage=$(($freemem / 512))

if [ "$usage" -gt "$threshold" ]

then

/etc/init.d/service_name restart

     if [ "$usage" -gt "$threshold2" ]

     then

     echo "The memory usage has reached $usage% on $HOSTNAME." | mail -s "High Memory Usage Alert" [email protected]


     fi
fi

Name this as alert.sh and execute the command: chmod +x alert.sh

Configure a cron to run this script every 10 minutes

Make sure to replace '512' with your server total memory in MB and '[email protected]' with actual email address. This will send an email alert whenever memory usage goes beyond 95 % and will restart the service "service_name" if it reaches 90%

2

You can use a shell script in cron with the free command to monitor the memory and act acording its values. For example, to monitor RAM memory:

#!/bin/bash

LOG_DIR=/var/log/memory_monitor.log

DATE=$(date +%d/%m/%Y)
TIME=$(date +%H:%M)
TIMESTAMP="$DATE $TIME"

MONITOR=$(free | grep Mem)
MEM_USED=$(echo $MONITOR | awk '{ print $3 }')
MEM_FREE=$(echo $MONITOR | awk '{ print $4 }')

echo "$TIMESTAMP $MEM_USED $MEM_FREE" >> $LOG_DIR

Instead of echoing the output, you could eval the values to the limits you want and mail, reboot or whatever action you want:

if [ eval_values > threshold ]
then
    # Do stuff (mail, reboot, etc)
fi

Then you add it to crontab to be run in the intervals you want.

1

another useful utility from the sysstat package is sar.

For memory information, use:

$ sar -r 1
Linux 3.0.0-12-generic (HFC-2600)       05/03/2012      _i686_  (4 CPU)

01:35:45 PM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact
01:35:46 PM    118484    907364     88.45     59200    317368   2169716    104.75    412476    436140

I could definitely use more RAM on this box.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .