1

I recently started renting a VPS, it's running Ubuntu Server 18.04.4 LTS on 8GB of RAM. I only ever used it to run a small (SpigotMC) Minecraft server. When the JRE for the server is not running all commands run perfectly well. I usually start the server in a tmux session like this:

java -Xms1G -Xmx4G -XX:+UseConcMarkSweepGC -jar spigot-1.15.2.jar

After I detach detach from the session basic bash commands and things like APT still work. But when try to look at any man page:

man: fork failed: Resource temporarily unavailable

Once in a blue moon I'm able to look at a man page, but it is rare. Also occasionally the server process itself experiences related issues. Sometimes in the logs I get an error like

com.mojang.authlib.GameProfile@deadbeef[id=<null>,name=example,properties={},leg
acy=false] (/123.456.789.0:55100) lost connection: Internal Exception: java.lang
.OutOfMemoryError: unable to create native thread: possibly out of memory or pro
cess/resource limits reached

or

[21:45:15] [Server-Worker-614/WARN]: Server-Worker-614 died
java.lang.OutOfMemoryError: unable to create new native thread

I cannot find it in the logs but once I also got an error directly mentioning pthread_create with the error code EAGAIN. Recently the whole server just crashed after a java.lang.OutOfMemoryError.

From my research this suggests that the server is either out of memory or a limit for the amount of threads has been reached. However, at the times of the crashes RAM usage was only between one and two GB. Additionally the entire system is always running less than 80 threads (I checked using ps -eLF | wc -l). I also read about the possibility of the stack running out, but I am not sure how to confirm that. And I would assume the consequences of that case would be more grave.

I'm especially perplexed because some things like apt still work, but man does not. Could it be an issue with the VPS-hosting provider? Even when the server is idling (i.e. no JRE running), the SSH sometimes lags for seconds. I'm at my wit's end, do you have any idea what might be the root cause of this issue?

Edit: As it turns out, my instance is running in an OpenVZ container. Here is /proc/user_beancounters:

Version: 2.5
       uid  resource                     held              maxheld              barrier                limit              failcnt
   *******: kmemsize                 36102144             40902656  9223372036854775807  9223372036854775807                    0
            lockedpages                     0                    0  9223372036854775807  9223372036854775807                    0
            privvmpages                414235               521970  9223372036854775807  9223372036854775807                    0
            shmpages                      278                 2064  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numproc                        69                   69                  400                  400                    0
            physpages                  373418              1191913              2097152              2097152                    0
            vmguarpages                     0                    0  9223372036854775807  9223372036854775807                    0
            oomguarpages               374566              1193699                    0                    0                    0
            numtcpsock                      0                    0  9223372036854775807  9223372036854775807                    0
            numflock                        5                   12  9223372036854775807  9223372036854775807                    0
            numpty                          2                    4  9223372036854775807  9223372036854775807                    0
            numsiginfo                      0                  138  9223372036854775807  9223372036854775807                    0
            tcpsndbuf                       0                    0  9223372036854775807  9223372036854775807                    0
            tcprcvbuf                       0                    0  9223372036854775807  9223372036854775807                    0
            othersockbuf                    0                    0  9223372036854775807  9223372036854775807                    0
            dgramrcvbuf                     0                    0  9223372036854775807  9223372036854775807                    0
            numothersock                    0                    0  9223372036854775807  9223372036854775807                    0
            dcachesize               18575360             18673664  9223372036854775807  9223372036854775807                    0
            numfile                       840                 1320  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numiptent                      44                   44                 2000                 2000                    0
  • 1
    What does systemd-detect-virt say about your server – is it an actual KVM-based or Xen-based VPS, or is it an OpenVZ container? – user1686 Apr 5 at 20:40
  • It says openvz. – grinsekrokodil Apr 6 at 0:06
  • Yeah, so it's a type that's often heavily overprovisioned and shares resources between customers (I wouldn't even call it a "VPS", it's on the same level as a Docker or LXC container)... Does your server have a /proc/user_beancounters file? Add its contents to your post – in OpenVZ, it lists additional limits that your hosting provider applies, such as maximum processes or threads. – user1686 Apr 6 at 6:53
  • I just thought that might be it, but I don't think it's the limiting fator. I'll add it. – grinsekrokodil Apr 6 at 12:15
  • Since I moved to my new VPS I got the same symptoms as @grinsekrokodil. I don't have a solution. But I noticed that I only get the Exception when running under a restricted user in tmux. My normal login user account (in and out tmux) and the restricted user account outside of tmux start the server just fine. – koala Jun 4 at 21:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.