0

Every night the web server is using 100% CPU and all websites stop working. I have checked the server logs, in the log, I am getting below error – and this process taking more than 100% CPU usage and load avg also increased.

Here the full log for my issue, after server load average goes to 100% and server went down.

[cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: * Server auth using Basic with user 'ae1973565bbf19a6c79f': /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: * Using Stream ID: 1 (easy handle 0x1dbb460): /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: > PUT /api/v1/customers/154 HTTP/2\r: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: Host: track.customer.io\r: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: Authorization: Basic XXXXXXXXXXXXXXXX=\r: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: Accept: */*\r: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: Content-Length: 61\r: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: Content-Type: application/x-www-form-urlencoded\r: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: \r: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: * We are completely uploaded and fine: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: < HTTP/2 401 \r: /usr/local/cpanel/cgi-sys/ea-php54
cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: < content-length: 56\r: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: < content-type: application/json; charset=utf-8\r: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: < date: Sat, 15 Jun 2019 22:33:54 GMT\r: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: < status: 401 Unauthorized\r: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: * Authentication problem. Ignoring this.: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: < www-authenticate: Basic realm="Customer.io edge event collection"\r: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: < via: 1.1 google\r: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: < alt-svc: clear\r: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 16373] [client yy.yy.yy.yy:5646] AH01215: < \r: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Expire in 9 ms for 1 (transfer 0x20cb460): /usr/local/cpanel/cgi-sys/ea-php54
cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Expire in 9 ms for 1 (transfer 0x20cb460): /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Expire in 16 ms for 1 (transfer 0x20cb460): /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Expire in 12 ms for 1 (transfer 0x20cb460): /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Expire in 12 ms for 1 (transfer 0x20cb460): /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Expire in 16 ms for 1 (transfer 0x20cb460): /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Expire in 16 ms for 1 (transfer 0x20cb460): /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Expire in 16 ms for 1 (transfer 0x20cb460): /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Expire in 32 ms for 1 (transfer 0x20cb460): /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Expire in 50 ms for 1 (transfer 0x20cb460): /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Expire in 50 ms for 1 (transfer 0x20cb460): /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Expire in 64 ms for 1 (transfer 0x20cb460): /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Expire in 50 ms for 1 (transfer 0x20cb460): /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Expire in 50 ms for 1 (transfer 0x20cb460): /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Expire in 64 ms for 1 (transfer 0x20cb460): /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Expire in 50 ms for 1 (transfer 0x20cb460): /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Expire in 50 ms for 1 (transfer 0x20cb460): /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Expire in 50 ms for 1 (transfer 0x20cb460): /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: *   Trying 35.227.225.220...: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * TCP_NODELAY set: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Expire in 200 ms for 4 (transfer 0x20cb460): /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Connected to track.customer.io (35.227.225.220) port 443 (#0): /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * ALPN, offering h2: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * ALPN, offering http/1.1: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * successfully set certificate verify locations:: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: *   CAfile: /etc/pki/tls/certs/ca-bundle.crt: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215:   CApath: none: /usr/local/cpanel/cgi-sys/ea-php54
[cgi:error] [pid 15569] [client xx.xx.xx.xx:42486] AH01215: * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256: /usr/local/cpanel/cgi-sys/ea-php54
```
Improve this question
12
  • 1
    Please make sure there is no XY problem here. What is your research so far?
    – Kamil Maciorowski
    Jun 25, 2019 at 12:37
  • Still, we did not find the exact issue, sometimes server load increased to 100 and all sites went down after we reboot the system and all working fine. So we need to kill the process PID which consuming the CPU
    – Bala
    Jun 25, 2019 at 12:41
  • 1
    Here's the answer you perhaps don't want to hear, but it's essential if you're maintaining a server which hosts a bunch of websites: Don't add a script to cron that regularly kills processes if they are above a certain amount of CPU usage. Please instead show us which process is causing the problems, then let's find out how to prevent them.
    – slhck
    Jun 25, 2019 at 13:05
  • @slhck, yes you are right and accept your answer. but this issue I am facing at a nigth time at that time no one is online, that's the problem and its test environment only.
    – Bala
    Jun 25, 2019 at 13:19
  • 1
    Even better when it's a test environment! You should then try to log the process CPU consumption over night: superuser.com/a/466121/48078. Collect some information for one day, then continue troubleshooting. If you could edit your question with this information and show what you've tried, others can help you.
    – slhck
    Jun 25, 2019 at 13:22

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .