Wget hangs in the middle of get request

Question

Sometimes wget or curl randomly hangs in the middle of get request in Ubuntu 10.10:

curl "http://api.indeed.com/ads/apisearch?publisher=7570038743238473"
wget "http://api.indeed.com/ads/apisearch?publisher=7570038743238473" -O - -d

Seems the problem is in some low level library that does the request, but not in headers, cookies or caches.

Any ideas?

Answer 1

This URL fetches without errors or hangs for me. As wget and curl are, generally, these low-level libraries, and they only share the HTTP/TCP protocol, the error can be in two places:

1. The request can be mishandled on your side. I've experienced symptoms like this several times, and looks like that they can be caused by a node on your route to the server which has MTU less than yours (often it is your provider's VPN, if they use it), and because of that packets get dropped (I don't know what happens there exactly, as they should be fragmented in theory).

   A quick check is to lower the MTU locally on your Ethernet/WiFi link like this:

   sudo ip link set $IFACE mtu 1300
   

   where $IFACE is your network interface (eth0, wlan0 or such).

2. The request can be staled by server. The server can either be overloaded, misconfigured or just blocking your requests selectively. This is the less probable variant, and it cannot be diagnosed easily.

Answer 2

My ideas:

• The server is occasionally overloaded.
• Use a sniffer (such as Wireshark) to see what is happening when it hangs.
• Use strace to see what wget/curl are doing when it hangs.

Answer 3

I'd use tcpdump to see where exactly it hangs. You might find regularities.