0

I have a server that receives large POST (100,000 chars max) requests and passes them back to a Glassfish4 application server using mod_jk. The requests are POST requests with a content-type of application/x-www-form-urlencoded. One of the parameters is a large XML document prepared on the hosting web page. I encode the xml using encodeURIComponent() on the page before adding to the parameters then send the POST using XmlHttpRequest.

Using RHEL 7 on Amazon EC2 machine with adequate resources. 

Free Command:
            total        used        free      shared  buff/cache   available
Mem:        3689604     2564636      712168       16992      412800      946260

ps -aux:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
apache    8276  1.6  1.6 293492 62228 ?        S    14:51   1:08 /usr/sbin/httpd
postfix   8326  0.0  0.1  91168  3868 ?        S    14:59   0:00 pickup -l -t un
apache    8372  0.0  0.1 237408  5896 ?        S    15:05   0:00 /usr/sbin/httpd
apache    8575  0.0  0.1 237504  6160 ?        S    15:24   0:00 /usr/sbin/httpd
apache    8595  6.5  3.3 354720 122936 ?       S    15:26   2:23 /usr/sbin/httpd
apache    8649  6.4  2.8 335576 103624 ?       S    15:32   1:57 /usr/sbin/httpd
apache    8655  7.0  2.9 342236 110316 ?       S    15:32   2:07 /usr/sbin/httpd
apache    8698  0.0  0.1 237400  5600 ?        S    15:37   0:00 /usr/sbin/httpd
apache    8700  0.0  0.1 237404  5512 ?        S    15:37   0:00 /usr/sbin/httpd
apache    8711  0.0  0.1 237408  5912 ?        S    15:41   0:00 /usr/sbin/httpd
apache    8714  0.0  0.1 237404  5384 ?        S    15:46   0:00 /usr/sbin/httpd
root      8716  0.0  0.0      0     0 ?        S    15:48   0:00 [kworker/0:1]
root      8719  0.0  0.0      0     0 ?        R    15:53   0:00 [kworker/0:0]

All works fine at low load but once the load increases moderately (about 1 request every second) some of the apache processes jump to high CPU and the VM CPU reaches 100%.

I've tried lots of things - upgraded apache from 2.2 to 2.4.16, turned off SSL, recompiled apache on a clean VM, tried closing connections on browser, but the problem keeps recurring.

My mod_status and ps -aux shows the processes that are offending and when I run an strace I get the following repeating constantly:

read(18, "\6\37\372", 3)                = 3
write(18, "\0224\0\0", 4)               = 4
read(18, "AB\0\3", 4)                   = 4
read(18, "\6\37\372", 3)                = 3
write(18, "\0224\0\0", 4)               = 4
read(18, "AB\0\3", 4)                   = 4
read(18, "\6\37\372", 3)                = 3
write(18, "\0224\0\0", 4)               = 4
read(18, "AB\0\3", 4)                   = 4
read(18, "\6\37\372", 3)                = 3
write(18, "\0224\0\0", 4)               = 4
read(18, "AB\0\3", 4)                   = 4
read(18, "\6\37\372", 3)                = 3
write(18, "\0224\0\0", 4)               = 4
read(18, "AB\0\3", 4)                   = 4
read(18, "\6\37\372", 3)                = 3
write(18, "\0224\0\0", 4)               = 4
read(18, "AB\0\3", 4)                   = 4
read(18, "\6\37\372", 3)                = 3
write(18, "\0224\0\0", 4)               = 4
read(18, "AB\0\3", 4)                   = 4
read(18, "\6\37\372", 3)                = 3
write(18, "\0224\0\0", 4)               = 4
read(18, "AB\0\3", 4)                   = 4

The only think I haven't tried is posting the XML text as text/plain with no parameters.

Can anyone think of anything here?

Thank you for your help.

Mike Russiello

Improve this question
3
  • The only other thing I'd add is that the glassfish processes seem to work just fine. It seems to be just the Apache process that is consuming CPU.
    – Mike Russiello
    Jul 20, 2015 at 16:11
  • All - In case you run into this, here's what I found that really helped troubleshooting. First, the server-status was showing the hung processes in "W" mode - write. Basically, the connection was waiting for the Glassfish AS to finish. On Glassfish, the reply timeout on the mod_jk connectors was 900 seconds, or about 15 minutes. So, the process just kept polling for that period and sucked major CPU. In workers.properties there was no reply_timeout value set. So, I set the worker.wokerx,reply_timeout to around 20 seconds and this had immediate effect.
    – Mike Russiello
    Jul 21, 2015 at 12:56
  • Since the issue only seemed to happen for a specific POST transaction, I created a special worker just for this process, which let me set the reply_timeout to a more lenient value for all other processes. Also, on glassfish, I set the timeout on the mod_jk connector to a lower value - it was 900 seconds - I set it to about 60 seconds.
    – Mike Russiello
    Jul 21, 2015 at 13:00

0

Reset to default

You must log in to answer this question.