I recently started renting a VPS, it's running Ubuntu Server 18.04.4 LTS on 8GB of RAM. I only ever used it to run a small (SpigotMC) Minecraft server. When the JRE for the server is not running all commands run perfectly well. I usually start the server in a tmux session like this:
java -Xms1G -Xmx4G -XX:+UseConcMarkSweepGC -jar spigot-1.15.2.jar
After I detach detach from the session basic bash commands and things like APT still work. But when try to look at any man page:
man: fork failed: Resource temporarily unavailable
Once in a blue moon I'm able to look at a man page, but it is rare. Also occasionally the server process itself experiences related issues. Sometimes in the logs I get an error like
com.mojang.authlib.GameProfile@deadbeef[id=<null>,name=example,properties={},leg
acy=false] (/123.456.789.0:55100) lost connection: Internal Exception: java.lang
.OutOfMemoryError: unable to create native thread: possibly out of memory or pro
cess/resource limits reached
or
[21:45:15] [Server-Worker-614/WARN]: Server-Worker-614 died
java.lang.OutOfMemoryError: unable to create new native thread
I cannot find it in the logs but once I also got an error directly mentioning pthread_create
with the error code EAGAIN
. Recently the whole server just crashed after a java.lang.OutOfMemoryError
.
From my research this suggests that the server is either out of memory or a limit for the amount of threads has been reached. However, at the times of the crashes RAM usage was only between one and two GB. Additionally the entire system is always running less than 80 threads (I checked using ps -eLF | wc -l
). I also read about the possibility of the stack running out, but I am not sure how to confirm that. And I would assume the consequences of that case would be more grave.
I'm especially perplexed because some things like apt
still work, but man
does not. Could it be an issue with the VPS-hosting provider? Even when the server is idling (i.e. no JRE running), the SSH sometimes lags for seconds. I'm at my wit's end, do you have any idea what might be the root cause of this issue?
Edit: As it turns out, my instance is running in an OpenVZ container. Here is /proc/user_beancounters
:
Version: 2.5
uid resource held maxheld barrier limit failcnt
*******: kmemsize 36102144 40902656 9223372036854775807 9223372036854775807 0
lockedpages 0 0 9223372036854775807 9223372036854775807 0
privvmpages 414235 521970 9223372036854775807 9223372036854775807 0
shmpages 278 2064 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
numproc 69 69 400 400 0
physpages 373418 1191913 2097152 2097152 0
vmguarpages 0 0 9223372036854775807 9223372036854775807 0
oomguarpages 374566 1193699 0 0 0
numtcpsock 0 0 9223372036854775807 9223372036854775807 0
numflock 5 12 9223372036854775807 9223372036854775807 0
numpty 2 4 9223372036854775807 9223372036854775807 0
numsiginfo 0 138 9223372036854775807 9223372036854775807 0
tcpsndbuf 0 0 9223372036854775807 9223372036854775807 0
tcprcvbuf 0 0 9223372036854775807 9223372036854775807 0
othersockbuf 0 0 9223372036854775807 9223372036854775807 0
dgramrcvbuf 0 0 9223372036854775807 9223372036854775807 0
numothersock 0 0 9223372036854775807 9223372036854775807 0
dcachesize 18575360 18673664 9223372036854775807 9223372036854775807 0
numfile 840 1320 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
dummy 0 0 9223372036854775807 9223372036854775807 0
numiptent 44 44 2000 2000 0
systemd-detect-virt
say about your server – is it an actual KVM-based or Xen-based VPS, or is it an OpenVZ container?openvz
./proc/user_beancounters
file? Add its contents to your post – in OpenVZ, it lists additional limits that your hosting provider applies, such as maximum processes or threads.