TL:DR: PHP/Apache in docker is continuously creating 'files' in
/proc/*/fd/ eventually making the site unusable.
We are running a PHP app in docker (
As far as I can tell this was caused by this error
[core:crit] [pid 17] (24)Too many open files: AH00529: /var/www/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable and that '/var/www/' is executable
So I went and checked what files where opened by the Apache processes.
lsof -a -p 15 output:
apache2 15 www-data 35 unknown /proc/15/fd/35 (readlink: Permission denied) apache2 15 www-data 37 unknown /proc/15/fd/37 (readlink: Permission denied) apache2 15 www-data 38 unknown /proc/15/fd/38 (readlink: Permission denied)
The fd number increments continuously when the website is accessed, until eventually it reaches the open file limit.
The reason for
(readlink: Permission denied) was a docker security feature, so all those handles are actually open TCP sockets.
apache2 15 www-data 198u sock 0,8 0t0 794575 protocol: TCP apache2 15 www-data 200u sock 0,8 0t0 795679 protocol: TCP apache2 15 www-data 201u sock 0,8 0t0 795681 protocol: TCP
However netstat lists no unusual open connections.
netstat -a output:
Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 127.0.0.11:42045 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN udp 0 0 127.0.0.11:59658 0.0.0.0:* Active UNIX domain sockets (servers and established) Proto RefCnt Flags Type State I-Node Path