0

I'm using wget in a script to download a changing file from Dropbox:

wget -4 -O filename.ext "https://www.dropbox.com/s/<fixed-hash-id>/filename.ext?dl=1"

I notice that wget resolves the IP address of www.dropbox.com, and it keeps changing quite often.

# Note: these examples are from multiple runs of wget, with duplicates removed:

Resolving www.dropbox.com... 108.160.165.139
Resolving www.dropbox.com... 108.160.165.147
Resolving www.dropbox.com... 108.160.165.20
Resolving www.dropbox.com... 108.160.166.13
Resolving www.dropbox.com... 108.160.166.142
Resolving www.dropbox.com... 108.160.166.148
Resolving www.dropbox.com... 108.160.166.20

After resolution, wget connects to the IP address and sends a request. At this point, Dropbox returns status 302 FOUND, and redirects wget to a CDN server like this:

Location: https://dl.dropboxusercontent.com/content_link/<random-generated-hash>?dl=1 [following]

Then, wget resolves the redirect to multiple IP addresses, for example:

Resolving dl.dropboxusercontent.com... 54.225.184.81, 107.22.170.202, 54.235.147.68, ...

It then connects to the IP addresses (attempting in linear order), gets 200 OK and starts downloading the file.

Now the problem is, quite often wget takes a while to resolve host names to IP addresses, and it has to do that twice. I tried to speed up the process by directly connecting to one of the known IP addresses, but that doesn't work.

  • If I directly connect to a known IP address of www.dropbox.com, I get status 301 Moved Permanently and get redirected to the URL with the host name instead (shown above). This effectively slows down the process further.

  • I attempted to reuse the CDN URL directly using either the host name or one of the known IP addresses of the CDN server, but they also seem to redirect (IP address to host name) or simply reject the request (404 Not Found).

Attempting to download the files using a web browser never has any such DNS resolution delays at all. What can I do to make wget cache DNS results or use some sort of DNS cache that the browsers seem to do?

Alternatives to wget or Dropbox are welcome. I tried curl but it too is plagued by the same problems.

  • You could always set up a caching DNS resolver on your computer. Something like bind or DNSmasq. This would cache results for the whole OS. – user2313067 Dec 10 '14 at 6:58
  • How would it handle 301 Moved Permanently (sites that keep changing their IP addresses and rejecting direct IP connections with a redirection to host names)? – ADTC Dec 10 '14 at 7:33
  • It wouldn't even be aware of it. You would simply query your own computer to resolve the domain name, and it would cache the reply. – user2313067 Dec 10 '14 at 7:35
  • You did not understand my question, and do note from my question that somehow Dropbox servers seem to know that I'm making a direct request to their IP address (skipping DNS resolution of their host name). They seem to detect this (I don't know how) and redirect/force me to do the DNS resolution anyway. Caching DNS resolution would be useless in this case because then I will keep hitting the same IP address and getting the same 301 Moved Permanently status with redirect request each time. My question is, what if the local cache (my own computer) is already outdated... then what? – ADTC Dec 10 '14 at 7:44
  • 1
    Dropbox has a TTL of 1 min on their DNS so unless you configure the resolver to ignore that (not sure it's possiblewith the most common), it would cache the IP for at most a minute. If you want to cache it more then that, you won't have any guarantees that it is still valid (though I doubt Dropbox changes IP that often). When you try accessing it directly by IP, the site receives a host header that is set to the IP and not the domain name, which is why it triggers a redirect. By using a caching DNS resolver, you will set the host header correctly and shouldn't be rejected by the server. – user2313067 Dec 10 '14 at 10:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.