This may seem programming related, but this is an OS question.

I'm writing a small high performance daemon that takes thousands of connections per second. It's working fine on Linux (specifically Ubuntu 9.10 on EC2). On Mac OS X if I throw a few thousand connections at it (roughly about 16350) in a benchmark that simply opens a connection, does it's thing and closes the connection, then the benchmark program hangs for several seconds waiting for a socket to become available before continuing (or timing out in the process).

I used both Apache Bench as well as Siege (to make sure it wasn't the benchmark application).

So why/how is Mac OS X limiting the RATE at which sockets can be used, and can I stop it from doing this?

Or is there something else going on?

I know there is a file descriptor limit, but I'm not hitting that. There is no error on accepting a socket, it's simply hangs for a while after the first (roughly) 16000, waiting -- I assume -- for the OS to release a socket. This shouldn't happen since all prior the sockets are closed at that point. They're supposed to come available at the rate they're closed, and do on Ubuntu, but there seems to be some kind of multi (5-10?) second delay on Mac OS X.

I tried tweaking with ulimit every-which-way. Nada.

  • Do not cross post - serverfault.com/questions/145907/…
    – MDMarra
    May 27 '10 at 15:37
  • this is a good question, and it's one of those gray areas that could be equally acceptable on Super User or Server Fault. still, you shouldn't crosspost first and ask questions later. post on one site or the other. then, if it doesn't get good responses after a reasonable time, you can flag for moderator attention to request it be migrated to the other site. May 27 '10 at 18:08
  • 1
    @MarkM Why not? It's applicable to both and though slightly more relevant to ServerFault I got the answer here and posted the answer to the other. It's even relevant to StackOverflow, although I decided not to post there since I was reasonably sure the problem was not in the code. Cross posting is not bad if it's relevant to each community since it provides value to both. You don't always know which angle the answer will come from.
    – pbhogan
    May 27 '10 at 18:13
  • @quack quixote I was not aware there was a protocol for this. I don't really agree with it but will try to follow it in the future.
    – pbhogan
    May 27 '10 at 18:19
  • 1
    generally crossposts are bad (because the questioner doesn't know to which site a question belongs; usually the result is question migrations and duplications). that's not the case here -- as i said, i think this one is equally acceptable on either site. still, i think crossposts are to be avoided. post-on-one-then-migrate-if-necessary seems like a reasonable protocol to me. thanks! May 27 '10 at 18:46

Mac OS X starts opening ephemeral ports at 49152. Port numbers are 16-bit unsigned integers, so there are 65535 possible ports. 65535 - 49152 = 16383. I think you've got 16K ports in TIME_WAIT.

Update: You might want to look at the following sysctl(8) variables:

net.inet.ip.portrange.lowfirst: 1023  
net.inet.ip.portrange.lowlast: 600  
net.inet.ip.portrange.first: 49152  
net.inet.ip.portrange.last: 65535  
net.inet.ip.portrange.hifirst: 49152  
net.inet.ip.portrange.hilast: 65535  

I think if you set hifirst to something lower, you'll increase the number of ephemeral ports available on your system.

There might be a socket option or something to tell the stack to basically violate the TCP spec and use a nonstandard value for TIME_WAIT, but I'm not enough of a Mac OS X sockets programmer to know that.

Update 2: You probably want to use setsockopt(2) to set SO_REUSEADDR.

  • You are correct. Netstat tells me they're all in TIME_WAIT. I'm using SO_REUSEADDR on the daemon already, but the benchmark apps probably don't, which is why it's keeling over. Of course it's highly unlikely a single client will be opening that many simultaneous connections anyway (as opposed to reusing one or a few connections).
    – pbhogan
    May 27 '10 at 16:58
  • sudo sysctl -w net.inet.ip.portrange.first=32768 helped alleviate the problem. On Mac OS X one probably needs to set both hifirst and first because they're treated the same and you don't know which one is being used. See also: ncftp.com/ncftpd/doc/misc/ephemeral_ports.html#FreeBSD
    – pbhogan
    May 27 '10 at 17:05
  • This answer absolutely solved similar problem I had, with Java/JMeter, 1000 requests/second to a fast web service (local perf testing).
    – StaxMan
    Jul 20 '13 at 4:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.