Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I recently stumbled upon this site which claims that the deep web is a tremendous section of the world wide web that isn’t accessible via regular queries through Yahoo, Google or various other search engines.

According to the site:

  • General public info on the deep Web is actually 4 hundred to 5 hundred times larger compared to generally characterized as World-wide-web
  • The deep Web consists of 6,500 terabytes of information in comparison to 20 terabytes of information that is accessible in the surface Web
  • The deep Web is made up of closely 550 billion unique records and documents in contrast to the 1 billion from the surface Web
  • 60 of the largest deep-Websites mutually consist of 750 terabytes of data — enough by themselves in order to surpass the size of the public Web by 40 times
  • Overall quality content material from the deep Web is 1,000 to 2,000 times better than that of the surface Web

Are the above claims true? Especially, the last one. If so, is there a way to search for content on the Deep web using a search engine like Google or by any other means?

share|improve this question

closed as not constructive by Oliver Salzburg, ChrisF, sblair, slhck, LawrenceC May 18 '12 at 14:32

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.If this question can be reworded to fit the rules in the help center, please edit the question.

AKA, Darknet, and No Google cannot crawl every server when they are password – Moab May 18 '12 at 17:55
up vote 6 down vote accepted

A lot of content is not searchable for the simple reason that it is created dynamically when requested, or is behind limited access in multitude of ways (pay walls, etc.). This is the original meaning of the phrasing "Deep web".

By its very definition, the "deep web" is not searchable. It is a term to describe the collection of material that is explicitly not searchable.

It is not, as some entities like to use the term, some "cool" part of the internet where "regular people" aren't allowed. I admit that "deep web" sounds cool in the same way as "black hole" does, but in practice that does not make any difference.

E.g. I run a personal web server. Some parts of it are restricted to real life acquaintances in different ways. Would it make sense for you to access it? Hint: No.

The statements you cite, e.g.

  • Overall quality content material from the deep Web is 1,000 to 2,000 times better than that of the surface Web

are just nonsense, really.

share|improve this answer

Almost all of those points are pure marketing and I don't beleve them, but there could me some small truth behind them. What is likely happening is two things:

  1. The search engine may not respect robots.txt (the short version is it is a file telling search engines not to include a page that you have on your website in the search results). So because it includes all web pages (not just the ones it was allowed to look at) it will have more search results.

  2. It searches tor for hidden services and lists them on the search results. These sites have .onion on the end of their domain name and you must use a tor proxy to access them.

Between those two methods I think I can see where they are getting their numbers from, but I don't see how that is "1,000 to 2,000 times better than that of the surface Web"

share|improve this answer

As I understand it, 'Deep Web' refers to the portion of the web that isn't easily indexed in an automatic manner; for example, many pages are dynamically generated on the fly in response to a query, or a form being filled in. Or, some sites require member login before all the content is available. These kinds of sites are not generally transparent to automated search robots, and so the only part of the site that ends up being indexed is the 'surface' portion.

The actual numbers are of course hard to prove; the Wikipedia article references this study which speculates at the size as being around 91,000 TB.

Recently Google announced that they're working to improve the way that their search bots 'read' web pages, which implies they're trying to index more of this hard-to-get-to content. Engadget has a writeup on it here.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged .