Apache will not start in Mountain Lion. Syntax OK. What is happening?

Question

Over the course of two days I was doing a lot of changes to my Apache config. At times changes did not seem to be picked up, and it was always the case that I had somehow messed up the syntax of the config file, something apachectl -t always would show. The syslog would also show error messages concerning respawning httpd.

The following day, I needed to make some additional changes, and noted they were not picked up by Apache. I wanted to see the errors in the apache error_log, but there was nothing to see. Removing them and restarting apache did nothing. The files were not even recreated!

So of course, I checked the syntax again, and apachectl -t spits out Syntax OK. Nothing wrong there, it seems. Then I check my syslog, and it is full of errors concerning httpd:

Mar 13 11:03:41 skinny com.apple.launchd[1] (org.apache.httpd[22707]): Exited with code: 1
Mar 13 11:03:41 skinny com.apple.launchd[1] (org.apache.httpd): Throttling respawn: Will start in 10 seconds

Stopping, starting, restarting with apachectl did absolutely nothing. But I was actually managing to access localhost. So what was happening?

Answer 1

Once I realized that I could actually access localhost on port 80, even though I had issued apachectl stop, the path to illumination was clear. I needed to find out what was actually occupying port 80, and kill it.

First I checked what was holding on to it

netstat -ta |grep LISTEN
tcp4       0      0  *.65374                *.*                    LISTEN     
tcp4       0      0  *.65373                *.*                    LISTEN     
tcp4       0      0  *.spytechphone         *.*                    LISTEN     
tcp4       0      0  *.blp1                 *.*                    LISTEN     
tcp4       0      0  *.8193                 *.*                    LISTEN     
tcp46      0      0  *.http                 *.*                    LISTEN     
tcp4       0      0  localhost.25986        *.*                    LISTEN     

This did not give me the PID, but at least showed me that it was in fact Apache that held on to port 80. Next I got the PIDs

$  sudo lsof -iTCP -sTCP:LISTEN
COMMAND     PID              USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
launchd       1              root   24u  IPv6 0x3917f1c4693aa457      0t0  TCP *:rfb (LISTEN)
launchd       1              root   25u  IPv4 0x3917f1c4693af43f      0t0  TCP *:rfb (LISTEN)
httpd     11654              root    5u  IPv6 0x3917f1c46be2d077      0t0  TCP *:http (LISTEN)
httpd     11655              _www    5u  IPv6 0x3917f1c46be2d077      0t0  TCP *:http (LISTEN)
httpd     11662              _www    5u  IPv6 0x3917f1c46be2d077      0t0  TCP *:http (LISTEN)
httpd     11663              _www    5u  IPv6 0x3917f1c46be2d077      0t0  TCP *:http (LISTEN)
httpd     11664              _www    5u  IPv6 0x3917f1c46be2d077      0t0  TCP *:http (LISTEN)
httpd     20907              _www    5u  IPv6 0x3917f1c46be2d077      0t0  TCP *:http (LISTEN)
httpd     20909              _www    5u  IPv6 0x3917f1c46be2d077      0t0  TCP *:http (LISTEN)
httpd     20910              _www    5u  IPv6 0x3917f1c46be2d077      0t0  TCP *:http (LISTEN)
httpd     20933              _www    5u  IPv6 0x3917f1c46be2d077      0t0  TCP *:http (LISTEN)

Using the PID I could now learn a bit about the processes before killing it

$  ps -ef 11654
  UID   PID  PPID   C STIME   TTY           TIME CMD
    0 11654     1   0  3:40PM ??         0:00.54 /usr/sbin/httpd -k start -e debug

And there it was. It seems that the Apache process I started the day before using debug flags was not being killed when issuing the stop command to apachectl. Therefore I had to manually kill it.

$  sudo killall httpd

This took care of the "zombie" (OK, not a zombie process in os theory, but still not wanting to die). Restarting apache now worked fine!