How to free a port on OS X when lsof(1) reports that it is not in use

Question

I am a developer working in OS X on Eclipse. A bug I have been working to fix in a web application results in some very strange behavior when resetting an instance of Tomcat, resulting in the following error message upon forcing a shutdown and restart of the server (it will not shutdown cleanly, probably due to the bug I am trying to fix so I need to kill it, both kill -9 <pid> and Eclipse's internal "force terminate" give the same behavior):

Several ports (23432, 34543) required by Tomcat v8.0 Server at localhost are already in use. The server may already be running in another process, or a system process may be using the port. To start this server you will need to stop the other process or change the port number(s).

This is fine, I've worked on this stuff a long time and usually it means Tomcat is still running somewhere, just need to remove it and things will work.

However, lsof(1) does not list any active processes holding onto that port:

user@yosemite ~ %  sudo lsof -Pan -i tcp -i udp | grep 23432
COMMAND    PID           USER   FD   TYPE             DEVICE SIZE/OFF   NODE NAME
launchd      1           root   23u  IPv6 0x57073763bfdd9c27      0t0    TCP *:5900 (LISTEN)
launchd      1           root   26u  IPv4 0x57073763bfddfb77      0t0    TCP *:5900 (LISTEN)
launchd      1           root   30u  IPv6 0x57073763bfdd9727      0t0    TCP [::1]:631 (LISTEN)
launchd      1           root   31u  IPv6 0x57073763bfdd9c27      0t0    TCP *:5900 (LISTEN)
launchd      1           root   32u  IPv4 0x57073763bfddf2a7      0t0    TCP 127.0.0.1:631 (LISTEN)
launchd      1           root   34u  IPv6 0x57073763bfdd9227      0t0    TCP *:22 (LISTEN)
launchd      1           root   37u  IPv4 0x57073763bfdde9d7      0t0    TCP *:22 (LISTEN)
launchd      1           root   41u  IPv4 0x57073763bfddfb77      0t0    TCP *:5900 (LISTEN)
launchd      1           root   47u  IPv4 0x57073763bfddf2a7      0t0    TCP 127.0.0.1:631 (LISTEN)
...
user@yosemite ~ %  sudo lsof -Pan -i tcp -i udp | grep 23432

I've verified that the ports are actually in use via a little python script, just to make sure that Eclipse isn't mistaken:

user@yosemite ~ % cat socket_open.py
#!/usr/bin/env python
import sys, traceback, socket

HOST=''
PORT = int(sys.argv[1])

sck = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print "Attempting to bind '%s':%s" % (HOST, PORT)
try:
    sck.bind((HOST, PORT))
    print sck.getsockname()
except Exception as exc:
    traceback.print_exc()

Simple enough, just try to bind to the same port and throw an Exception if it doesn't work.

user@yosemite ~ % sudo python socket_open.py 23432
Password:
Attempting to bind '':23432
Traceback (most recent call last):
  File "socket_open.py", line 10, in <module>
    sck.bind((HOST, PORT))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 48] Address already in use

There is a zombie (java) process floating around with the PID of the Tomcat that I originally killed, and I suspect this fellow has something to do with it:

user@yosemite ~ % ps wwwaux | grep java | grep -v 'grep'
user          975   0.0  0.0        0      0   ??  ?E   11:15AM   0:00.00 (java)

Is there any way I can free up the ports that are in use without rebooting or logging out?

This happens every single time I recreate my bug and it is a huge time sink to have to stop and reboot on every debugging pass.

On Linux I would probably run gdb -p PID and close(<fdnum>) but on OS X I can't even find what (if any) file descriptor refers to that address.

Answer 1

I suppose netstat is not working either?

I had this same thing happening to me on Linux, but a long time ago (I'm pretty sure it was kernel 2.0.36). The port was hooked on my eth1. What I did was to disable the network adapter and re-enable it a couple of seconds later. That 'cleared' the kernel structures that held the port in its "half-open" state.

I don't know if you can issue a down on the loopback interface and have it work, but maybe it's worth a try.

You could try sending a SIGCHLD to launchd and see what happens. Or even a SIGHUP to tell it to reinit and (possibly) reap zombie children (kill -HUP 1, kill -s HUP 1.

Another trick to try would be to "wrap" the Tomcat instance with a launch script that would run the true Tomcat and remain waiting for it to close. Then you could see what happens when the wrapper script itself gets killed, or you could try and listen for the rogue java process to ensure it doesn't become a zombie.