1

I would like to know the size of the documents hosted on a website.

A solution could be to download all contents of this website with tools such as wget -r. However, this specific website hosts many huge files and the amount of data probably exceed 3To. I cannot afford to use such bandwidth and host so many data.

A second solution could be to connect by ftp to this website. But I don't have ftp access to this server.

Is there another solution to my problem? To sum up, I want to know how big is a directory hosted on a website and I have only read and browse permissions.

P-S: I'm using Linux (Debian) and welcome in particular every solution compatible with it.

1
  • 2
    You can use HTTP HEAD requests and look at the Content-Length header in the response, but not all web servers include that piece of information, particularly for dynamic content. You'd also need to know which exact URLs to ask for.
    – user
    Nov 18, 2015 at 10:59

2 Answers 2

1

As noted, you could use the Content-Length item from an HTTP header. Various tools can get the header information, e.g.,

$ lynx -dump -head http://localhost
HTTP/1.1 200 OK
Date: Wed, 18 Nov 2015 11:21:40 GMT
Server: Apache
Last-Modified: Sun, 10 Nov 2013 20:02:38 GMT
ETag: "bf2-4ead818dfd380"
Accept-Ranges: bytes
Content-Length: 3058
MS-Author-Via: DAV
Content-Type: text/html
Connection: close

However, you will still have to retrieve a given web page to extract URLs from it. Lynx gives a list at the end of a dump, e.g,.

$ lynx -dump http://localhost
...
References

   1. http://localhost/NetBeans-ROR-plugin/updates/updates.xml
   2. http://localhost/webcal
   3. http://localhost/changepassword
   4. http://localhost/profilemanager
   5. http://www.apple.com/server/

Putting those together, it would be possible to write a script that retrieved URLs from a page, then walked through the list of referenced URLs, first doing a "head" (probably faster than reading the whole referenced page), inspecting both the Content-Length and Content-Type to see if that information is available, and if so, whether it points to another web page to crawl through.

For reference:

Since you only want sizes, your script could simply use the header information when it points to non-HTML items where the content-length is known.

0

Use curl with -I option to get only the response header (i.e. without file downloading)

$ curl -sI REMOTE_FILE_LOCATION | grep Content-Length

or if you want to extract only the number

$ curl -sI REMOTE_FILE_LOCATION | grep Content-Length | cut -d' ' -f2

curl is usually installed by default on Linux, so you don't have to install it. In order to use lynx from the previous answer, you'll have to download and install it.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .