Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

This all started when I was trying to download a podcast on my Mac. This happens in both Snow Leopard and Lion when the Ethernet interface is set to use Jumbo Frames.

prompt> curl -v -o x.mpg http://audio.wnyc.org/freakonomics_specials/freakonomics_specials041112.mp3 
* About to connect() to audio.wnyc.org port 80 (#0)
*   Trying 204.93.180.53... Local Interface en0 is ip 172.16.1.2 using address family 2
* Local port: 0
* connected
* Connected to audio.wnyc.org (204.93.180.53) port 80 (#0)
> GET /freakonomics_specials/freakonomics_specials041112.mp3 HTTP/1.1
> User-Agent: curl/7.19.7 (universal-apple-darwin10.0) libcurl/7.19.7 OpenSSL/0.9.8r zlib/1.2.3
> Host: audio.wnyc.org
> Accept: */*
> 
OK
< Server: nginx/0.7.65
< Date: Mon, 16 Apr 2012 23:39:03 GMT
< Content-Type: audio/mpeg
< Content-Length: 42075070
< Last-Modified: Tue, 10 Apr 2012 21:15:08 GMT
< Connection: close
< Content-Disposition: attachment
< Accept-Ranges: bytes
< 
  0 40.1M    0     0    0     0      0      0 --:--:--  0:00:24 --:--:--     0^C

The headers come through fine, but the download never gets anywhere. This is the only web server I have this kind of trouble with, but it's still annoying and I'd like to see if there's a fix other than forgoing jumbo frames everywhere.

I determined that I can download a partial file as long as the size of the chunk I'm downloading is 1448 bytes or less. I can use range 0-1447 or 10000-11447 so it's not the position in the file, it's the size of the chunk. The WAN MTU on my router had been set to 1500, so I tried reducing it in steps until I got to 1400 and it still didn't make any difference.

I was thinking this was a problem with Path MTU Black Hole discovery because the problem goes away when I stop using jumbo frames on the ethernet interface. But I have everything set up for black hole discovery (as far as I can tell) and ping doesn't see any problem:

ping  -g1435 -G1445 204.93.180.53PING 204.93.180.53 (204.93.180.53):
(1435 ... 1445) data bytes
1443 bytes from 204.93.180.53: icmp_seq=0 ttl=51 time=69.223 ms
1444 bytes from 204.93.180.53: icmp_seq=1 ttl=51 time=75.542 ms
1445 bytes from 204.93.180.53: icmp_seq=2 ttl=51 time=72.136 ms
1446 bytes from 204.93.180.53: icmp_seq=3 ttl=51 time=73.732 ms
1447 bytes from 204.93.180.53: icmp_seq=4 ttl=51 time=72.057 ms
1448 bytes from 204.93.180.53: icmp_seq=5 ttl=51 time=73.377 ms
1449 bytes from 204.93.180.53: icmp_seq=6 ttl=51 time=71.717 ms
1450 bytes from 204.93.180.53: icmp_seq=7 ttl=51 time=73.293 ms
1451 bytes from 204.93.180.53: icmp_seq=8 ttl=51 time=71.874 ms
1452 bytes from 204.93.180.53: icmp_seq=9 ttl=51 time=73.206 ms
1453 bytes from 204.93.180.53: icmp_seq=10 ttl=51 time=71.289 ms

In fact, ping works all the way up to 1494 bytes, though I believe the Mac counts the bytes differently than other *nixes. (I think it counts as data the 8 bytes of the ICMP header, but not the 20 bytes of IP header, unlike most which don't count either and some which count both.)

I don't want to give up the performance benefits of jumbo frames on my LAN just for this one broken web site, but of course I want my podcast. So I'm looking for suggestions for things to try.

My Mac has 2 built-in ethernet ports, so one thing I tried was connecting the second one with MTU 1500 and forcing curl to use that port. Curl said it was using that port, but the MTU of that port had no effect on whether the download worked or not. It was the MTU of the first active ethernet port that mattered. I don't know what that means, either.

Suggestions, anyone?

share|improve this question
    
That is weird, as if it is MTU related, then the MTU choke point is your router, not the remote website, so it shouldn't make any difference. Can you run a tcpdump and see if you get an icmp unreachable for sites that work and those that don't? –  Paul Apr 17 '12 at 2:57
    
@Paul, I don't know, it could be due to a misconfiguration on the nginx host or the route between it and the file server or something like that. Some part of the server cluster sees jumbo frames being used and sends the file data in a jumbo frame, but some other part has Don't Fragment set so the data never makes it out. That's my best guess as to what's happening. –  Old Pro Apr 17 '12 at 3:03
    
But if the router is mtu 1500 how would a remote server ever see a jumbo frame? Unless one of the session setup packets from your machine isn't getting there because it is DF and >1500 - I can't imagine how a http GET could get that big. tcpdump/wireshark is needed I think, to narrow it down. –  Paul Apr 17 '12 at 3:07
    
@Paul, the Maximum Segment Size is negotiated during the TCP setup. Switching jumbo frames off on my Mac drops my advertised MSS from 8960 to 1460. –  Old Pro Apr 17 '12 at 5:19
    
Ah of course. MSS get negotiated before the PMTU kicks in and indicates to your machine that the destination can't handle jumbo frames. Your end will most likely bring the MSS down in response to PMTU, but the other end is not doing the same, so they have the existing MSS to work with. My guess is they are blocking PMTU but are using jumbo frames themselves internally. Can you clamp MSS just to this destination on OSX? –  Paul Apr 17 '12 at 5:34

1 Answer 1

up vote 2 down vote accepted

Workarounds:

  • Lower the interface MTU to 1500. System Preferences -> Network, select interface (e.g. Built-in Ethernet), click Advanced... button, Ethernet tab, MTU drop-down or set Configure to Automatically. This solves the problem, but means you cannot use Jumbo Frames on the LAN (at least from this computer).
  • If this were on Linux: use iptables to clamp the TCP MSS for just this host. --set-mss 1460

Solution:

  • Fix the server configuration to either lower the MTU or make Path MTU Discovery work.

As it happens, I was finally able to find someone to talk to at the CDN company who could implement the solution, and they lowered the MTU and that solved the problem.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.