2

I have my own server running lighttpd. When viewing the headers with "curl -I ..." on my laptop through my standard / regular internet connection, I get this:

    HTTP/1.1 200 OK
Content-Type: application/zip
ETag: "546653951"
Last-Modified: Wed, 08 May 2013 15:35:42 GMT
Content-Length: 28166067
Date: Wed, 08 May 2013 19:07:36 GMT
Server: lighttpd/1.4.28

When I switch my laptop over to my cell phone connection (wifi hotspot), I run the exact same command in the exact same terminal to the exact same server, I get back this:

    HTTP/1.1 200 OK
Content-Type: application/zip
Accept-Ranges: bytes
ETag: "546653951"
Last-Modified: Wed, 08 May 2013 15:35:42 GMT
Content-Length: 28166067
Date: Wed, 08 May 2013 19:09:23 GMT
Server: lighttpd/1.4.28

Note that "Accept-Ranges: bytes" is present in the second case but not the first.

What can possibly be causing this? I desperately need this pause / resume capability, it's been missing on my connection for as long as I can remember and just never investigated why (not just to my own server, but to ANY server / file that I wanted to download)... From a different computer I have access to, running the same curl command shows that Accept-Ranges: bytes is present, so I'm assuming there is something wonky with my regular ISP here at home.

Would network hardware cause this? Incompatible router / switch maybe? Or would it be the ISP itself?

Any thoughts?


As requested by Dennis, here is the output:

    echo > tempfile; wget -d -c -O tempfile redtwitz.com
Setting --continue (continue) to 1
Setting --output-document (outputdocument) to tempfile
DEBUG output created by Wget 1.13.4 on linux-gnu.

URI encoding = `UTF-8'
--2013-05-10 12:20:48--  http://redtwitz.com/
Resolving redtwitz.com (redtwitz.com)... 184.22.37.72
Caching redtwitz.com => 184.22.37.72
Connecting to redtwitz.com (redtwitz.com)|184.22.37.72|:80... connected.
Created socket 4.
Releasing 0x00000000013d1310 (new refcount 1).

---request begin---
GET / HTTP/1.1
Range: bytes=1-
User-Agent: Wget/1.13.4 (linux-gnu)
Accept: */*
Host: redtwitz.com
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... 
---response begin---
HTTP/1.1 200 OK
Date: Fri, 10 May 2013 16:21:56 GMT
Server: Apache/2.2.14 (Ubuntu)
Last-Modified: Thu, 02 Aug 2012 13:41:17 GMT
ETag: "a819c40-d-4c64890da1940"
Content-Length: 13
Vary: Accept-Encoding
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/html

---response end---
200 OK
Registered socket 4 for persistent reuse.
Length: 13 [text/html]
Saving to: `tempfile'

100%[================================================================>] 13          --.-K/s   in 0s      

2013-05-10 12:20:49 (783 KB/s) - `tempfile' saved [13/13]

more tempfile

edtwitz.com
7
  • What's the exact command you're using?
    – Dennis
    May 8 '13 at 19:38
  • I don't want to publicly state my server here, but the exact command would look like "curl -I myserver.com/myfile.zip". So just "curl -I" and then the url to the file I would download.
    – user85116
    May 8 '13 at 20:49
  • The Accept-Ranges header by itself is irrelevant. What happens if you simply specify a range (e.g., curl --range 0-100 URL)?
    – Dennis
    May 8 '13 at 20:54
  • It's not irrelevant at all... it shows that something very odd is going on between the 2 ISP connections. If the same header information was missing on both connection you would be right (irrelevant), but there is obviously something odd going on here. Executing curl --range 5-10 URL gives me the entire file on my regular ISP, but gives me the requested 6 bytes on my cellular ISP.
    – user85116
    May 10 '13 at 13:15
  • Irrelevant by itself, i.e., if partial downloads worked. Could you execute these commands and post the output? echo > tempfile; wget -d -c -O tempfile redtwitz.com
    – Dennis
    May 10 '13 at 15:40
2

RFC2616 "Hypertext Transfer Protocol -- HTTP/1.1", section 14.5:

14.5 Accept-Ranges The Accept-Ranges response-header field allows the server to indicate its acceptance of range requests for a resource:

 Accept-Ranges     = "Accept-Ranges" ":" acceptable-ranges
 acceptable-ranges = 1#range-unit | "none"

Origin servers that accept byte-range requests MAY send

 Accept-Ranges: bytes

but are not required to do so. Clients MAY generate byte-range requests without having received this header for the resource involved. Range units are defined in section 3.12.

Servers that do not accept any kind of range request for a resource MAY send

 Accept-Ranges: none

to advise the client not to attempt a range request.

In short, that's the remote server telling your UA that it's willing to accept a request for only part of the resource in question, as described in RFC2616 section 14.35. Since this is the mechanism by which resumption of a failed download is implemented, seeing the Accept-Ranges header in the server response is actually a good sign that you'll be able to accomplish your goal here.

Indeed, curl appears to implement this capability, as described here; the two forms given of the command are:

cat file-that-failed-to-download.zip | curl -C - http://www.somewhere.com/file-I-want-to-download.zip >successfully-downloaded.zip

and

curl -C - -o partially_downloaded_file 'www.example.com/path/to/the/file'

It looks to me as though both forms should behave more or less identically, but I haven't tried either so can't be sure. Assuming Curl behaves as advertised, you should be able simply to reissue the same command (possibly with slight modification, as in the first form where you'd need to change the filenames) each time you need to resume the download, and Curl will examine what you've downloaded so far and issue a Range header specifying only the remaining bytes in the file.

As for why you aren't seeing the Accept-Ranges header in the response to your original request, perhaps there's some statefulness on the part of the server, such that it recognizes the second request from your UA for the same resource as such, and helpfully includes the Accept-Ranges header to make sure your UA is aware that an attempt to resume the download will succeed. In any case, it shouldn't matter particularly; per the RFC cite above, your client may (in the absence of a pre-existing Accept-Ranges: none header from the server) issue a byte-range request whether it's already seen the Accept-Ranges header or not, and indeed wouldn't need to specify a range on the first request in any case, since it's attempting to download the entire resource rather than just part of it.

5
  • Thanks Aaron for your comments. But at this point I'm not even downloading anythinng, there are no partial files... I'm just fetching and displaying the http headers; that's it! I can toggle between the 2 network connections 10 times, executing "curl -I myfile.zip" 5 times with each connection... the 5 times with my real ISP I get no Accept-Ranges, the 5 times with my blackberry connection I do get it.
    – user85116
    May 8 '13 at 20:55
  • Perhaps there's some difference in the headers as sent from the two connections, such that the one sent via cellular includes a header that prompts the server to send an Accept-Ranges in response. Either way, though, the effect should be the same: calling curl as described in the page linked from my answer should resume the download, as I gather from your question you're looking to accomplish -- as described in the RFC, the Accept-Ranges header is advisory only (except where its value is none), so your UA doesn't need to see it first before sending a request for partial content. May 8 '13 at 20:59
  • Ignoring the header and then just attempting to download a byte range does not work on my regular ISP; curl ends up downloading the entire file. But switching to my cell ISP does give the correct byte range...
    – user85116
    May 10 '13 at 13:16
  • Well, that's just weird. It seems unlikely, too, that your ISP or network hardware should be filtering only those headers from HTTP traffic, and nothing else; is it possible your lighttpd instance is in some way misconfigured to produce this result? A longshot, I know, but it seems at least a little likelier than the other. May 10 '13 at 15:23
  • I tried running same test with a different server in a different country; same result as my own server. The fact that I can't pause / resume any downnload at all using downloader tools makes me think the problem is on my end, not the server end...
    – user85116
    May 10 '13 at 16:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.