Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I've been wondering recently about how the infrastructure of the Internet really works.

I know that I have an Internet Service Provider (ISP) that supplies my connection to the Internet.

But what I don't know is: Who provides the Internet to the ISP? And who supplies it to them? Is there a never-ending loop that eventually connects us all together?

share|improve this question
74  
Turtles all the way down, as they say –  Reid Mar 11 '12 at 1:01
7  
We're all connected... in the Great Circle of Life. –  Iszi Mar 11 '12 at 8:25
7  
I thought this would have been a lame question to ask.. turns out I was wrong. –  Power-Inside Mar 11 '12 at 9:52
13  
I hope this does not get closed. It is an interesting question, and far from trivial (unlike some people seem to think). –  sleske Mar 11 '12 at 16:04
14  
Remember, it's about the "spirit of the law," not the "letter of the law" sometimes. This question is a little bit on the open-ended/not-a-problem side, but seems reasonable, has a real question which can be answered, and is well-written. Seems reasonable to me. –  nhinkle Mar 13 '12 at 20:46
show 4 more comments

4 Answers

up vote 306 down vote accepted

How do I figure out the Internet's infrastructure?

Let's suppose we don't know about the history of the Internet, nor do we have access to any online resources that explain us this. Then, the only way to learn how the Internet infastructure is built is to go back to the roots. Using existing protocols to discover how our Internet is built.

Specifically, the Internet Control Message Protocol or ICMP defines the Echo request and the Echo reply. By increasing the Time To Live of IP packets by 1 each iteration, you can find each next hop on the path to your target. This allows you to get a list of hops between you and your target, the classic traceroute.

On Windows, you can use tracert; on Linux and Mac OS X, you can use traceroute.

So, let's do a traceroute from Belgium to the United States; Stack Exchange looks like a good target.

Tracing route to stackexchange.com [64.34.119.12] over a maximum of 30 hops:

  ... redacted ...

  5    10 ms    12 ms    12 ms  te-3-3.car2.Brussels1.Level3.net [212.3.237.53]
  6    11 ms    11 ms    15 ms  ae-0-11.bar2.Brussels1.Level3.net [4.69.148.178]
  7    20 ms    13 ms    15 ms  ae-7-7.ebr1.London1.Level3.net [4.69.148.182]
  8    16 ms    16 ms    18 ms  vlan101.ebr2.London1.Level3.net [4.69.143.86]
  9    83 ms    84 ms    87 ms  ae-44-44.ebr1.NewYork1.Level3.net [4.69.137.78]
 10    84 ms    93 ms    97 ms  ae-71-71.csw2.NewYork1.Level3.net [4.69.134.70]
 11    87 ms    96 ms    83 ms  ae-2-70.edge1.NewYork1.Level3.net [4.69.155.78]
 12    84 ms    93 ms    84 ms  gig2-0.nyc-gsr-b.peer1.net [216.187.123.5]
 13    87 ms    84 ms    85 ms  gwny01.stackoverflow.com [64.34.41.58]
 14    87 ms    82 ms    87 ms  stackoverflow.com [64.34.119.12]

Interesting, we now know that Belgium, London and New York are all connected to Level3. Level3 can be seen as an ISP to ISPs, they simply interconnect multiple ISPs. Here is a picture of how it's connected:

Let's go the opposite direction, China! The first thing I could find is the search engine Baidu.

Tracing route to baidu.com [123.125.114.144] over a maximum of 30 hops:

  ... redacted ...

  5    12 ms    10 ms    12 ms  ae0.anr11.ip4.tinet.net [77.67.65.177]
  6   167 ms   167 ms   167 ms  xe-5-1-0.sjc10.ip4.tinet.net [89.149.185.161]
  7   390 ms   388 ms   388 ms  as4837.ip4.tinet.net [77.67.79.150]
  8   397 ms   393 ms   397 ms  219.158.30.41
  9   892 ms     *      392 ms  219.158.97.13
 10   407 ms   403 ms   403 ms  219.158.11.197
 11   452 ms   451 ms   452 ms  219.158.15.5
 12     *      434 ms   434 ms  123.126.0.66
 13   449 ms   450 ms   450 ms  61.148.3.34
 14   432 ms   433 ms   431 ms  202.106.43.66
 15   435 ms   435 ms   436 ms  123.125.114.144

Well, not much information about the Chinese ISPs there but we at least found Tinet. Here is a nice picture of their site that shows how they connect with the various ISPs:

They simply have a cloud of hops spread about the relevant part of the world they serve, and at the end points they connect to the ISPs. The reason they have a cloud of hops is for reliability, for when some hops fall out...

If you repeat this a few times, you can get an idea of how everything is connected.

So, what Network Tiers are there?

The huge networks we found through trace-routing are known as Tier 1 networks.

Although there is no authority that defines tiers of networks participating in the Internet, the most common definition of a tier 1 network is one that can reach every other network on the Internet without purchasing IP transit or paying settlements.

By this definition, a tier 1 network is a transit-free network that peers with every other tier-1 network. But not all transit-free networks are tier 1 networks. It is possible to become transit-free by paying for peering or agreeing to settlements.

Common definitions of tier 2 and tier 3 networks:

  • Tier 2: A network that peers with some networks, but still purchases IP transit or pays settlements to reach at least some portion of the Internet.

  • Tier 3: A network that solely purchases transit from other networks to reach the Internet.

If you click through to Tier 1 networks from the Internet Backbone page you get to a list of the current Tier 1 networks:

  • AT&T from USA
  • Centurylink (formerly Qwest and Savvis) from USA
  • Deutsche Telekom AG from Germany
  • Inteliquent (formerly Tinet) from USA
  • Verizon Business (formerly UUNET) from USA
  • Sprint from USA
  • TeliaSonera International Carrier from Sweden
  • NTT Communications from Japan
  • Level 3 Communications from USA
  • Tata Communications from India

It is not known if AOL Transit Data Network (ATDN) is still a Tier 1 network.

Wait, what... What is Peering?

These networks connect to each other through a process known as 'peering'. Most traffic needs to go over at least 2 different top tier networks in order to reach its destination, and the networks are bridged with peering arrangements. The way this usually works is that each party to the agreement will commit to routing x amount of traffic for the other party on their network, and vice-verse. There is usually no money exchanged in these arrangements, unless one side is sending or receiving a lot more data than the other sides.

Large companies can also go out and arrange their own peering relationships. For example Netflix has arranged its own peering and network infrastructure directly with multiple tier-1 networks so that its traffic is both cheaper and closer to end users on each of the popular US broadband ISP's.

See this Wikipedia page on Peering.

There's a lot more to read at those pages; this answer gives a general idea, discovering all the details are left as an exercise to the reader. You can ask questions about this subject in the comments below...

share|improve this answer
46  
Perhaps this would be suitable as a blog post as well? –  Ivo Flipse Mar 11 '12 at 9:51
8  
Because keeping it short and simple was never an option :D Excellent answer. +1 –  Doomsknight Mar 11 '12 at 15:21
    
holy shi-woah. Excellent write-up. –  tombull89 Mar 12 '12 at 21:13
2  
Perhaps it's useful to mention Internet Exchange Points ? You do state that many Tier-2 providers have peering arrangements, but don't mention how they physically do that. –  MSalters Mar 13 '12 at 15:16
2  
In summary, for the US: The phone company. –  Joshua Drake Mar 13 '12 at 17:42
show 4 more comments

Basically as Tom described. To put his words into a generality, there are several internet backbones that connect with each other. ISPs work with these internet backbones and then sell bandwidth to their users. No one really owns the internet as it comprises millions of servers everywhere. However, there are central hubs (backbones) that can cause huge interruptions to several people's internet connectivity to some websites and servers. Basically, if a backbone in Colorado goes down, it could prevent East Coast US users to lost connectivity to servers on the West Coast. However, they would still have access to their own East Coast servers (assuming that the traffic isn't routed through Colorado in this example).

If you're wondering why a webpage may not be loading and there are other people complaining about the same things in your area then I would look at the internet health reports.

I just took this snapshot. It looks like there is high latency between Cogent and AT&T which could affect some users from accessing some sites or delay the responses.

enter image description here

SouthPark picture thrown in as related topic fun.

enter image description here

share|improve this answer
42  
It was from the episode, "Over Lodging", where the internet went down. No one had internet access and no one could find the solution. The image above was the "backbone" of the internet that was causing the problem. South Park then made a mockery of it by making the backbone look like a Linksys WRT54G Router. One of the boys unplugged it and then plugged it back in. Internet was restored. –  kobaltz Mar 11 '12 at 15:02
add comment

The Internet is a collection of networks. Most networks have to pay something to send or receive data to other networks. This includes the network in your home, even if the "network" only has 1 computer. Tier 1 networks do not have to pay to send or receive data with each other; the fact that they are not paying for privilege of sending/receiving traffic is the main thing that (unofficially) defines what is a Tier 1 network.

Note that the legal agreements that these Tier 1 players have with each other usually specify that any network they allow to send/receive data with them for free has to agree not to send too much more than they receive.

share|improve this answer
    
I believe the phrase you were looking for in your first sentence was, "A series of tubes." –  Nick Stauner Feb 14 at 6:31
add comment

protected by nhinkle Mar 13 '12 at 20:48

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.