90

Occasionally, when downloading a file in a web browser, the download progress doesn't "know" the total size of the file, or how far along in the download it is -- it just shows the speed at which it's downloading, with a total as "Unknown".

Why wouldn't the browser know the final size of some files? Where does it get this information in the first place?

2

3 Answers 3

119

To request documents from web servers, browsers use the HTTP protocol. You may know that name from your address bar (it may be hidden now, but when you click the address bar, copy the URL and paste it in some text editor, you'll see http:// at the beginning). HTTP is a simple text-based protocol. It works like this:

First, your browser connects to the website's server and sends a URL of the document it wants to download (web pages are documents, too) and some details about the browser itself (User-Agent etc). For example, to load the main page on the SuperUser site, http://superuser.com/, my browser sends a request that looks like this:

GET / HTTP/1.1
Host: superuser.com
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.0 Safari/537.36
Accept-Encoding: gzip,deflate,sdch
Accept-Language: pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4
Cookie: [removed for security]
DNT: 1
If-Modified-Since: Tue, 09 Jul 2013 07:14:17 GMT

The first line specifies which document the server should return. The other lines are called headers; they look like this:

Header name: Header value

These lines send additional information that helps the server decide what to do.

If all is well, the server will respond by sending the requested document. The response starts off with a status message, followed by some headers (with details about the document) and finally, if all is well, the document's content. This is what the SuperUser server's reply for my request looks like:

HTTP/1.1 200 OK
Cache-Control: public, max-age=60
Content-Type: text/html; charset=utf-8
Expires: Tue, 09 Jul 2013 07:27:20 GMT
Last-Modified: Tue, 09 Jul 2013 07:26:20 GMT
Vary: *
X-Frame-Options: SAMEORIGIN
Date: Tue, 09 Jul 2013 07:26:19 GMT
Content-Length: 139672

<!DOCTYPE html>
<html>
    [...snip...]
</html>

After the last line, SuperUser's server closes the connection.

The first line (HTTP/1.1 200 OK) contains the response code, in this case it's 200 OK. It means that the server has decided it can return a document, as requested, and promises that the contents that follow will be such a document. If this is not the case the code will be something else, and it will provide some indication of the reason the server is not just returning a document as the response: for instance, if it cannot find the requested document, it is supposed to return 404 Not Found, and if you are not allowed to access the content in question it is supposed to return 403 Forbidden.

After this first status line, the response headers follow; they provide more information about the content being returned, such as its Content-type.

Next is an empty line. It signals the fact that no more response headers will follow. Everything past that line is the content of the document it requested. So in the above example, <!DOCTYPE html> is the first line of the SuperUser home page (a HTML document). If I was requesting a document to download, it would probably be some gibberish characters, because most document formats are unreadable without prior processing.

Back to headers. The most interesting one for us is the last one, Content-Length. It informs the browser how many bytes of data it should expect after the empty line, so basically it's the document size expressed in bytes. This header isn't mandatory and may be omitted by the server. Sometimes the document size can't be predicted (for example when the document is generated on the fly), sometimes lazy programmers don't include it (quite common on driver download sites), sometimes websites are created by newbies who don't know of such a header.

Anyway, whatever the reason is, the header can be missing. In that case the browser doesn't know how much data the server is going to send, and thus displays the document size as unknown, waiting for the server to close the connection. And that's the reason for unknown document sizes.

8
  • also will be omitted when being gzipped on the fly Jul 9, 2013 at 13:31
  • 5
    A very, very minor note: Browsers support protocols other than HTTP. But other protocols are rare these days, and essentially the same concepts apply to other protocols even though the details are different. Jul 9, 2013 at 13:57
  • 5
    @Thomas That is my experience these days. It’s been several years since I remember seeing an ftp URL in my browser. A few years ago I was using ftp—directly rather than with a browser—at work (almost entirely uploads), but those tasks are handled by scp now. The only thing I use ftp for today is uploading content to a minimalist web host. Of course, YMMV. ^_^ Jul 9, 2013 at 15:16
  • 3
    This is exactly the kind of answer that makes me love this site. How do I grant it a bounty? Jul 9, 2013 at 15:23
  • 1
    @ruda.almeida of you disagree with that, you can post about it on meta.superuser.com, it will be discussed and maybe someone will reopen the question.
    – gronostaj
    Jul 14, 2013 at 6:54
55

The HTTP Content-Length header is optional in some cases, and as such it may not be transmitted with the file; the end of the file will be signaled when the socket is closed.

1
  • 2
    To be precise, HTTP 1.0 defined the content length by closing the socket after each document. This is still supported in HTTP 1.1 for compatibility. But HTTP 1.1 allows to reuse connections for multiple documents if either the Content-Length header field is used or the document is transferred with Transfer-Encoding: chunked. The latter allows to dynamically generate content and send it piecewise as it gets generated and is able to signal the end of the document.
    – x4u
    Jul 9, 2013 at 17:21
4

When the content (e.g. a .pdf document or an Excel sheet) is created on the fly the size cannot be known before. In this cases the server cannot send you the size of the download before and the brower cannot display the total size.

10
  • 9
    @alfo going to have to disagree... if I'm streaming video, or even if I'm streaming any sort of data that's not a fixed size, if the point is to get the data to the user as quickly as possible, I won't know the size at the point where I start the transmittal
    – Foon
    Jul 9, 2013 at 13:01
  • 4
    @Alfo You can create data like .pdf files on the fly. As long as the data is not competely written you don't know the size but you can send the ata already to the browser. I've done this already in Java and sent an Excel file to the browser which as generated on the fly. From the browsers side it looked like a download but from the servers side it is a streaming. So it is possible to stream .pdf files even if you cnnot imagine this. From the browser it looks like a download without known length.
    – Uwe Plonus
    Jul 9, 2013 at 13:32
  • 8
    @Alfo - it only needs to be finished being created before the last packet is sent to the client. Jul 9, 2013 at 14:13
  • 5
    @Alfo I have never taked about video steaming but about streaming in general, which can also be streaming a .pdf file or an Excel sheet!
    – Uwe Plonus
    Jul 9, 2013 at 14:33
  • 3
    @Alfo - You have a valid point, dynamic files could be entirely created first in memory and then sent via HTTP and its easy to calculate content length. However, if the server is sending many large dynamically created files that will be broken up into many packets, it makes sense for the server to just start sending the chunks as they are calculated (versus to having to create every large file in memory and then send it). HTTP 1.1 specifically designed the chunked transfer encoding for this purpose.
    – dr jimbob
    Jul 9, 2013 at 16:31

Not the answer you're looking for? Browse other questions tagged .