3

TL:DR: PHP/Apache in docker is continuously creating 'files' in /proc/*/fd/ eventually making the site unusable.

We are running a PHP app in docker (php:5.6-apache). We deployed this stack to production a few weeks ago. After a few days costumers had problems accessing the site, which was caused by random 403 errors for various files, like javascript, image, etc.
As far as I can tell this was caused by this error

[core:crit] [pid 17] (24)Too many open files: AH00529: /var/www/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable and that '/var/www/' is executable

So I went and checked what files where opened by the Apache processes. lsof -a -p 15 output:

apache2  15 www-data   35   unknown                      /proc/15/fd/35 (readlink: Permission denied)
apache2  15 www-data   37   unknown                      /proc/15/fd/37 (readlink: Permission denied)
apache2  15 www-data   38   unknown                      /proc/15/fd/38 (readlink: Permission denied)

The fd number increments continuously when the website is accessed, until eventually it reaches the open file limit.

EDIT: The reason for (readlink: Permission denied) was a docker security feature, so all those handles are actually open TCP sockets.

apache2  15 www-data  198u     sock    0,8      0t0 794575 protocol: TCP
apache2  15 www-data  200u     sock    0,8      0t0 795679 protocol: TCP
apache2  15 www-data  201u     sock    0,8      0t0 795681 protocol: TCP

However netstat lists no unusual open connections. netstat -a output:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.11:42045        0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN
udp        0      0 127.0.0.11:59658        0.0.0.0:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   Path
  • Never seen a LAMP stack fail like this. Please check this Stack Overflow answer for some insight. – JakeGould Aug 20 '18 at 16:10
  • @JakeGould Problem is that it never closes the file handles so increasing the limit would only delay the issue. – Mormund Aug 20 '18 at 16:16
  • The problem is I have never seen a PHP server behave this way. The reality is you seem to have a coding/engineering problem if you have streams and files open for that long. PHP is not inherently designed to work this way. Thus you are hitting these walls. The only solution I can see is you need to close file handles more often. But without seeing your code, no other advice. And posting your code is impractical; what I am looking at here is a complete refactoring of PHP code or moving processes to a different language. – JakeGould Aug 20 '18 at 16:56
2

Turns out we had the xdebug extension installed which doesn't seem to close its sockets correctly.

socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 306
20    fcntl(306, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
20    connect(306, {sa_family=AF_INET, sin_port=htons(9000), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EINPROGRESS (Operation now in progress)
20    select(307, [306], [306], [306], {tv_sec=0, tv_usec=200000}) = 2 (in [306], out [306], left {tv_sec=0, tv_usec=199998})
20    getpeername(306, 0x7ffe1d6453f0, [16]) = -1 ENOTCONN (Transport endpoint is not connected)20
  • 1
    Good detective work! Check out this answer here for more details. XDebug is a valuable tool for non-production servers, so disabling it is a good idea in this case. But still it is nice to know what might solve this issue past outright disabling it. – JakeGould Aug 20 '18 at 20:05

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.