wget in a script to download a changing file from Dropbox:
wget -4 -O filename.ext "https://www.dropbox.com/s/<fixed-hash-id>/filename.ext?dl=1"
I notice that
wget resolves the IP address of
www.dropbox.com, and it keeps changing quite often.
# Note: these examples are from multiple runs of wget, with duplicates removed: Resolving www.dropbox.com... 184.108.40.206 Resolving www.dropbox.com... 220.127.116.11 Resolving www.dropbox.com... 18.104.22.168 Resolving www.dropbox.com... 22.214.171.124 Resolving www.dropbox.com... 126.96.36.199 Resolving www.dropbox.com... 188.8.131.52 Resolving www.dropbox.com... 184.108.40.206
wget connects to the IP address and sends a request. At this point, Dropbox returns status
302 FOUND, and redirects
wget to a CDN server like this:
Location: https://dl.dropboxusercontent.com/content_link/<random-generated-hash>?dl=1 [following]
wget resolves the redirect to multiple IP addresses, for example:
Resolving dl.dropboxusercontent.com... 220.127.116.11, 18.104.22.168, 22.214.171.124, ...
It then connects to the IP addresses (attempting in linear order), gets
200 OK and starts downloading the file.
Now the problem is, quite often
wget takes a while to resolve host names to IP addresses, and it has to do that twice. I tried to speed up the process by directly connecting to one of the known IP addresses, but that doesn't work.
- If I directly connect to a known IP address of
www.dropbox.com, I get status
301 Moved Permanentlyand get redirected to the URL with the host name instead (shown above). This effectively slows down the process further.
- I attempted to reuse the CDN URL directly using either the host name or one of the known IP addresses of the CDN server, but they also seem to redirect (IP address to host name) or simply reject the request (
404 Not Found).
Attempting to download the files using a web browser never has any such DNS resolution delays at all. What can I do to make
wget cache DNS results or use some sort of DNS cache that the browsers seem to do?
wget or Dropbox are welcome. I tried
curl but it too is plagued by the same problems.