1

Like if I was going to be someplace remote and I needed Wikipedia in its entirety or Stack Overflow in its entirety, is there a way to see how large a specific web domain is in GB?

This link is to a device that connects with satellites and provides a limited provision of data to an at-home hotspot. I would want to be able to hold an entire website to assist in cutting down on the data usage from that kind of device.

2
  • I don't think google indexes the size of individual pages, or if they do, they don't appear to make this information available publicly. You may be able to do this with a program designed for mass-downloading of web sites. Nov 28, 2014 at 2:03
  • It's also not something you can really measure. I mean, if there are 10,000 pages and each page contains the same header and footer, do you count the headers and footers? Because they'd probably only be stored once. Nov 28, 2014 at 2:05

3 Answers 3

0

Wikipedia offers a data dump that is in the terabytes of data.

For StackExchange, according to the torrent, the data dump is 22.6GB for Sep 2014.

Other sites may offer a way to download the data besides scraping the web pages. If you are going to scrape a dynamic web site, you can sometimes be caught in infinite loops that means the data you download may never end. Unless you are going for specific sets of information, or know that the web site is entirely static (only actual .HTML pages), you can not determine in an easy how big the site will be.

1

is it possible to determine through the internet or google, how large a website is

No.

Assuming you mean in advance of, or instead of, downloading everything onto your own computer and counting it there.

There is no general well-established convention or method by which administrators of websites make this information available.

You can work this out for some sites (see other answers) but the way you do this, and the probable success, vary from site to site.

or Google

They'd be in the best position to provide a general mechanism for all sites but there are complications

  • robots.txt exclusions that prevent Google seeing some or all of some sites
  • dynamic sites that can generate almost unlimited HTML content from limited resources (e.g. parametric search, some junk advertising sites at parked domains)
  • unclear site boundaries - some sites serve content from multiple domains.

Unless Google can see a way to monetise the provision of the information you seek, it may not be a high priority for them to provide it.

0

Lantern would only broadcast a specific subset of websites - Its essentially UUCP reinvented, so the idea you have to check the sizes likely won't work, or be necessary. Its one way so you shouldn't have to worry about the size of the data> its a broadcast medium like old fashioned TV or radio, rather than a two way medium like the internet. So, the main reason you're asking about this... well isn't going to be an issue. Wikipedia sums up outernet pretty well

Its an interesting problem though, so I'll give it a shot

The only way I can think of, that might work with some webpages is to use wget (with --spider so you don't download the page and --server-response to get the reported filesize). This approach has been heavily inspired by this SO question

This works with say superuser.com

[geek@phoebe os store]$ wget -v4 --spider --server-response superuser.com
Spider mode enabled. Check if remote file exists.
--2014-11-28 17:26:35--  http://superuser.com/
Resolving superuser.com (superuser.com)... 198.252.206.16
Connecting to superuser.com (superuser.com)|198.252.206.16|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Cache-Control: public, no-cache="Set-Cookie", max-age=60
  Content-Length: 71913
  Content-Type: text/html; charset=utf-8
  Expires: Fri, 28 Nov 2014 09:27:35 GMT
  Last-Modified: Fri, 28 Nov 2014 09:26:35 GMT
  Vary: *
  X-Frame-Options: SAMEORIGIN
  Set-Cookie: prov=85f6f157-7e84-43bf-b762-003cf7d8ff71; domain=.superuser.com; expires=Fri, 01-Jan-2055 00:00:00 GMT; path=/; HttpOnly
  Date: Fri, 28 Nov 2014 09:26:34 GMT
Length: 71913 (70K) [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

[geek@phoebe os store]$ wget -v4 --spider --server-response http://superuser.com/questions/845893/is-it-possible-to-determine-through-the-internet-or-google-how-large-a-website/845895#845895
Spider mode enabled. Check if remote file exists.
--2014-11-28 17:26:43--  http://superuser.com/questions/845893/is-it-possible-to-determine-through-the-internet-or-google-how-large-a-website/845895
Resolving superuser.com (superuser.com)... 198.252.206.16
Connecting to superuser.com (superuser.com)|198.252.206.16|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Cache-Control: public, no-cache="Set-Cookie", max-age=60
  Content-Length: 69163
  Content-Type: text/html; charset=utf-8
  Expires: Fri, 28 Nov 2014 09:27:43 GMT
  Last-Modified: Fri, 28 Nov 2014 09:26:43 GMT
  Vary: *
  X-Frame-Options: SAMEORIGIN
  Set-Cookie: prov=7d270174-a377-4758-bbff-f4c87054de67; domain=.superuser.com; expires=Fri, 01-Jan-2055 00:00:00 GMT; path=/; HttpOnly
  Date: Fri, 28 Nov 2014 09:26:42 GMT
Length: 69163 (68K) [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

But not, say

Google.com

[geek@phoebe os store]$ wget -v4 --spider --server-response google.com        
Spider mode enabled. Check if remote file exists.
--2014-11-28 17:29:06--  http://google.com/
Resolving google.com (google.com)... 74.125.68.113, 74.125.68.138, 74.125.68.100, ...
Connecting to google.com (google.com)|74.125.68.113|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 302 Found
  Cache-Control: private
  Content-Type: text/html; charset=UTF-8
  Location: http://www.google.com.sg/?gfe_rd=cr&ei=YkB4VMT6F9iDoAO2tIH4Dw
  Content-Length: 262
  Date: Fri, 28 Nov 2014 09:29:06 GMT
  Server: GFE/2.0
  Alternate-Protocol: 80:quic,p=0.02
Location: http://www.google.com.sg/?gfe_rd=cr&ei=YkB4VMT6F9iDoAO2tIH4Dw [following]
Spider mode enabled. Check if remote file exists.
--2014-11-28 17:29:06--  http://www.google.com.sg/?gfe_rd=cr&ei=YkB4VMT6F9iDoAO2tIH4Dw
Resolving www.google.com.sg (www.google.com.sg)... 74.125.68.94
Connecting to www.google.com.sg (www.google.com.sg)|74.125.68.94|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Fri, 28 Nov 2014 09:29:06 GMT
  Expires: -1
  Cache-Control: private, max-age=0
  Content-Type: text/html; charset=ISO-8859-1
  Set-Cookie: PREF=ID=a1dfee7d97d41db1:FF=0:TM=1417166946:LM=1417166946:S=Uzy6MmaLU-UegGZU; expires=Sun, 27-Nov-2016 09:29:06 GMT; path=/; domain=.google.com.sg
  Set-Cookie: NID=67=C_dkB1z4qdwwPkNMS80Ek1km-G4y716Evvh2BCEjYpdkpIJSAfXpjpTnSF496UlahPirO0Go-VhVxQjHlsEI_Hf4AxB9IfTyrGFzduyMB4rdTI-nX-kh0hlKhKQCrFg7; expires=Sat, 30-May-2015 09:29:06 GMT; path=/; domain=.google.com.sg; HttpOnly
  P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
  Server: gws
  X-XSS-Protection: 1; mode=block
  X-Frame-Options: SAMEORIGIN
  Alternate-Protocol: 80:quic,p=0.02
  Transfer-Encoding: chunked
Length: unspecified [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .