2

I’m running a Linux server that shows weird patterns in memory usage that I’m trying to investigate.

During normal operation, around 200MB of memory are used for applications according to munin, but every now and then (maybe every couple of days), there’s a sudden increase by around 1GB, as visible in this graph:

munin memory graph showing sudden increase in memory usage

I’ve been trying to find the cause of this, but I’m not sure what it could be. The fact that munin attributes this memory to “apps” suggests to me that one of the running processes is simply using a lot of memory, but looking at top when this happens shows that none of the processes uses nearly as much memory; in fact, the top 15 memory consuming processes don’t change at all when this occurs and the top one only uses 2.2% of the memory. Also, the memory info from ps doesn’t nearly add up to this much memory, but rather something in the ballpark of 200MB.

The height of the increase is usually the same, and as visible in this example, it just disappears after a few hours.

I’ve found that a reliable way to trigger this behaviour is by downloading a large file (1.4G) using wget via an ssh tunnel over the network to the local disk (network activity alone doesn’t trigger it).

To see the memory usage in more detail, I’ve started to log /proc/meminfo every 10 seconds and made a tarball with graphs from this data. The download causing the behaviour in question started at 10:35:32 and was finished at 10:38:53. The original log data is also included in the tarball.

In /proc/meminfo, there’s no value that goes up enough to explain that huge decrease of free memory (this is essentially what munin attributes to “apps”), which leaves me clueless. The amount of used memory that munin reports is also reported by free and top, but there are no processes in the process list that would explain such high memory usage, or the sudden increase when downloading a file.

What I’m trying to understand is:

  • Is it normal that the memory usage behaves in this way without any particular process using the increased memory?
  • What could be other sources of memory that’s not accounted for in /proc/meminfo?

Or could this be a sign of the system being compromised?

  • If you suspect a malicious file, I suggest an audit, verify every file on the server should be there. Although I question if its even possible to hide a running process. – Ramhound Jan 25 '14 at 17:46
1

munin is great for the graphs, but it doesn't tell which application caused the load.

One could use atop - advanced top.

atop records raw log file, and than one can open such a file and interactively go back and forth in time and see what was happening at some specific moment.

ator -r /var/log/atop_oame_date than using tand T keys you can go back to the time of the suspicious activity and see what was happening there.

  • atop is a good hint, I was looking for such a tool during debugging, thanks! Though it doesn’t help here because from watching top in real time when it happened, it seems that none of the processes in the process list is responsible for the suddenly increased memory usage. – Leon Weber Jan 26 '14 at 0:40
0

Your system is under no physical memory pressure. So when you try to transfer a 1.4GB file, it uses 1.4GB of memory. Your system has two choices -- it can either waste memory entirely or use it inefficiently. It chooses the latter. This is normal behavior.

  • Just, why would it use the memory? What would it use it for? I could understand if it was for buffers or filesystem caches, but the memory usage isn’t in either of these categories – it’s in the “apps” category. I’m trying to understand if “apps” can mean something else that running userspace processes. – Leon Weber Jan 27 '14 at 12:34
  • The memory holds the information that has been, or will be, transferred. There is a running userspace process performing the transfer. It's not a cache because it is mapped into memory by a running process. (The pages are, however, clean and discardable. So if the OS has anything better it can use the memory for, it will.) When the process terminates, the data will likely stay in memory as cache. But while the process has the mapping, it is "charged" to the process. When a system is under zero physical memory pressure, its memory usage is basically meaningless. – David Schwartz Jan 27 '14 at 12:36
  • Shouldn’t that be accounted for in /proc/meminfo in a category like {In,}Active(file) or Mapped? Judging from the graphs, it is not, in this case. – Jonas Schäfer Jan 27 '14 at 12:41
  • @JonasWielicki It depends on exactly what the application does. Since there's no pressure, nobody cares because it makes no difference. – David Schwartz Jan 27 '14 at 12:44
  • The application was wget, and it terminated after three minutes. Not even while wget was running were 1.4G attributed to the process, according to top. Also, if the data stays in the memory as cache, wouldn’t it be in munin’s “caches” graph? – Leon Weber Jan 27 '14 at 12:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.