EDIT: At first I thought this would be a function of how many browser tabs you are opening, and therefore you'd be suffering from "normal" functioning of your TCP/IP stack - rate-limiting of new TCP connections (and running out of available ephemeral ports). But I re-read your original post and noticed that you said SYN_SENT connections are piling up. This means that your machine IS managing to create lots of connections at once. I'll leave the old text of my original answer intact (if for no other reason than that I spent hours researching it, and it's pretty interesting stuff) but I just wanted to add this edit at the start to avoid reader-frustration. The end of my post covers router limitations, and I suspect this is the problem. You could try the same test on a different internet connection (3G using an iPhone's Personal Hotspot, for example).
Original post...
I just did an experiment - I used Chrome's Developer Tools window to monitor a page-load, whilst loading www.ft.com. I then went through every single line on the Network tab of the Developer Tools window and hand-copied each unique host/server accessed when loading www.ft.com's main page. In total, 71 unique servers were accessed:
ft.com
s1.ft-static.com
navigation.webservices.ft.com
s4.media.ft.com
s2.ft-static.com
im.ft-static.com
www.googleadservices.com
www.ft-static.com
amch.questionmarket.com
reg.ft-static.com
service.maxymiser.net
personalisation.ft.com
pong.qubitproducts.com
track.ft.com
stats.ft.com
www.googletagservices.com
js.revsci.net
cdn.krxd.net
partner.googleadservices.com
admin.brightcove.com
mostpopular.sp.ft-static.com
googleads.g.doubleclick.net
fastft.ftdata.co.uk
widget-cdn.rpxnow.com
static.chartbeat.com
pubads.g.doubleclick.net
www.google.com
4235225.fls.doubleclick.net
ads.rubiconproject.com
ping.chartbeat.net
pagead2.googlesyndication.com
c.brightcove.com
ads.revsci.net
media.ft.com
pix04.revsci.net
beacon.krxd.net
secure.fastclick.net
leadback.advertising.com
ib.adnxs.com
api.adsymptotic.com
optimized-by.rubiconproject.com
cdn.quilt.janrain.com
s1.test.ft-static.com
www.facebook.com
b.scorecardresearch.com
tap2-cdn.rubiconproject.com
b.scorecardresearch.com
im.test.ft-static.com
clamo.ftdata.co.uk
assets.rubiconproject.com
tap.rubiconproject.com
x.bidswitch.net
rp.gwallet.com
rc.d.chango.com
i.w55c.net
tags.bluekai.com
rbp.mxptint.net
ads.p161.net
magnetic.t.domdex.com
ads.mediade.sk
ad.turn.com
video.ft.com
pool.adizio.com
cdn.trn.com
p.brilig.com
s.ixiaa.com
api.bizographics.com
metrics.brightcove.com
goku.brightcove.com
www.google-analytics.com
brightcove.vo.llnwd.net
It gets worse though! Note that many of those hosts were accessed multiple times, but let's assume the web browser (or OS) is intelligent enough to keep TCP connections open for a few seconds and reuse them. Using lsof -n -p 8420 | grep -c "IPv4"
(Chrome is using PID 8420, and I have no other pages open in Chrome; I cleared the cache completely and disabled AdBlock etc beforehand). Whilst reloading the page, I simply upArrowed on the lsof
command repeatedly, with the above piped grep
giving me a count, i.e.:
$ lsof -n -p 8420 | grep -c "IPv4"
42
$ lsof -n -p 8420 | grep -c "IPv4"
64
$ lsof -n -p 8420 | grep -c "IPv4"
75
$ lsof -n -p 8420 | grep -c "IPv4"
85
$ lsof -n -p 8420 | grep -c "IPv4"
111
$ lsof -n -p 8420 | grep -c "IPv4"
129
$ lsof -n -p 8420 | grep -c "IPv4"
127
$ lsof -n -p 8420 | grep -c "IPv4"
128
$ lsof -n -p 8420 | grep -c "IPv4"
129
$ lsof -n -p 8420 | grep -c "IPv4"
128
$ lsof -n -p 8420 | grep -c "IPv4"
128
$
As you can see, loading a single web page resulted in a peak of 129 TCP connections. Whether or not they were in ESTABLISHED or some closing state like CLOSE_WAIT is irrelevant - those source-ports are not able to be used at that point and will (by default) take 60-480 seconds to be returned to the pool (RFC793 originally specified a MSL (Max Segment Lifetime) of 4 minutes but I think nowadays 60-120 seconds is the default). In Windows XP and Vista there were only a few thousand ephemeral ports available (by default), so as you can see, it would be very easy for your machine to run out of available ports. *It is important to note that as you can see from the first lsof
execution, my system already had 42 connections open, so this web page has resulted in 87 new connections being opened. I ran it a couple of times to ensure other apps (email client, etc) weren't temporarily boosting those numbers during my testing.
There are also other limits - I can only speak about Linux here as I haven't had time to research this on Windows (I've already spent a few hours looking into this problem and I need to eat!) ... but in Linux (and OSX) there is a limit on the maximum number of file descriptors per process, and that limit is set in a couple of places. The limit is 256. Bear in mind some web browsers launch one process per tab, so I'm not sure if you would be hitting this wall, but it would be easy enough to change the limits and re-test. Google for "ulimit max file descriptors" and "launchctl maxfiles" to find both places where it is set. Also see this Stackoverflow post.
I know from my pre-OSX days that Windows XP had a rate-limit on the creation of new TCP connections. This was to reduce the spreading of worms like Sasser, by limiting how rapidly they could create new TCP connections, but it was also to prevent OS resources being swallowed up - each new TCP connection requires resources which are usually be PRE-allocated by the OS during boot-up (to allocate/destroy buffers every time a TCP connection is opened/closed, would take longer, so the OS creates a connection table in advance). If you look at the configuration page for Torrent clients (such as Transmission), their default global limit for new TCP connections is 120, with a limit-per-torrent of 60. Their user manual recommends sticking to 120, and I've seen my web-browsing performance really take a dive if I set it to say 200-240. It's the TCP MSL that really limits things here - the time it takes for a TCP port in TIME_WAIT
to be returned to the pool.
My gut says that you're being limited by the fact that each of your browser tabs will be creating a lot of new TCP connections, and by loading say 15 tabs at once, you are asking the OS to open a lot of TCP connections all at the same time. When you re-run your test, ensure you wait at least 4 minutes between test runs, and use netstat -n -p tcp
(in Windows just use netstat -no
). If you're using *nix, use lsof -n -i | grep -c
beforehand to see how many connections your computer already has open.
LASTLY... check your router/gateway. I've seen a lot of BusyBox-based ADSL routers in the past that have a max limit of 1024 concurrent TCP connections (in any state, i.e. established, close_wait, etC). One client running a P2P download was enough to grind the router completely to a halt - and the symptom was exactly as you describe, but even if you only tried to load one web page. 1024 connections could be eaten up within minutes by p2p simply because all you have to do is open and close 1024 connections within 4 minutes and no new connections would be allowed. The common complaint was "when I run eMule, even with a slow download speed limit, everyone in my home complains that the internet becomes almost unusable". The reason a home internet router gives a s*** about TCP connections is because they almost always run NAT/PAT and often have stateful/SPI firewalls - both of which require the tracking of all connections. That said, routers made in the last few years should be better at handling this.
Hope that helps.
netstat -p tcp -n
and post the results here?