I've been wondering recently about how the infrastructure of the Internet really works.

I know that I have an Internet Service Provider (ISP) that supplies my connection to the Internet.

But what I don't know is: Who provides the Internet to the ISP? And who supplies it to them? Is there a never-ending loop that eventually connects us all together?

  • 91
    Turtles all the way down, as they say – Reid Mar 11 '12 at 1:01
  • 9
    We're all connected... in the Great Circle of Life. – Iszi Mar 11 '12 at 8:25
  • 10
    I thought this would have been a lame question to ask.. turns out I was wrong. – Irfan Mar 11 '12 at 9:52
  • 17
    I hope this does not get closed. It is an interesting question, and far from trivial (unlike some people seem to think). – sleske Mar 11 '12 at 16:04
  • 16
    Remember, it's about the "spirit of the law," not the "letter of the law" sometimes. This question is a little bit on the open-ended/not-a-problem side, but seems reasonable, has a real question which can be answered, and is well-written. Seems reasonable to me. – nhinkle Mar 13 '12 at 20:46
up vote 371 down vote accepted

How do I figure out the Internet's infrastructure?

Let's suppose we don't know about the history of the Internet, nor do we have access to any online resources that explain us this. Then, the only way to learn how the Internet infastructure is built is to go back to the roots. Using existing protocols to discover how our Internet is built.

Specifically, the Internet Control Message Protocol or ICMP defines the Echo request and the Echo reply. By increasing the Time To Live of IP packets by 1 each iteration, you can find each next hop on the path to your target. This allows you to get a list of hops between you and your target, the classic traceroute.

On Windows, you can use tracert; on Linux and Mac OS X, you can use traceroute.

So, let's do a traceroute from Belgium to the United States; Stack Exchange looks like a good target.

Tracing route to stackexchange.com [] over a maximum of 30 hops:

  ... redacted ...

  5    10 ms    12 ms    12 ms  te-3-3.car2.Brussels1.Level3.net []
  6    11 ms    11 ms    15 ms  ae-0-11.bar2.Brussels1.Level3.net []
  7    20 ms    13 ms    15 ms  ae-7-7.ebr1.London1.Level3.net []
  8    16 ms    16 ms    18 ms  vlan101.ebr2.London1.Level3.net []
  9    83 ms    84 ms    87 ms  ae-44-44.ebr1.NewYork1.Level3.net []
 10    84 ms    93 ms    97 ms  ae-71-71.csw2.NewYork1.Level3.net []
 11    87 ms    96 ms    83 ms  ae-2-70.edge1.NewYork1.Level3.net []
 12    84 ms    93 ms    84 ms  gig2-0.nyc-gsr-b.peer1.net []
 13    87 ms    84 ms    85 ms  gwny01.stackoverflow.com []
 14    87 ms    82 ms    87 ms  stackoverflow.com []

Interesting, we now know that Belgium, London and New York are all connected to Level3. Level3 can be seen as an ISP to ISPs, they simply interconnect multiple ISPs. Here is a picture of how it's connected:

Let's go the opposite direction, China! The first thing I could find is the search engine Baidu.

Tracing route to baidu.com [] over a maximum of 30 hops:

  ... redacted ...

  5    12 ms    10 ms    12 ms  ae0.anr11.ip4.tinet.net []
  6   167 ms   167 ms   167 ms  xe-5-1-0.sjc10.ip4.tinet.net []
  7   390 ms   388 ms   388 ms  as4837.ip4.tinet.net []
  8   397 ms   393 ms   397 ms
  9   892 ms     *      392 ms
 10   407 ms   403 ms   403 ms
 11   452 ms   451 ms   452 ms
 12     *      434 ms   434 ms
 13   449 ms   450 ms   450 ms
 14   432 ms   433 ms   431 ms
 15   435 ms   435 ms   436 ms

Well, not much information about the Chinese ISPs there but we at least found Tinet. Here is a nice picture of their site that shows how they connect with the various ISPs:

They simply have a cloud of hops spread about the relevant part of the world they serve, and at the end points they connect to the ISPs. The reason they have a cloud of hops is for reliability, for when some hops fall out...

If you repeat this a few times, you can get an idea of how everything is connected.

So, what Network Tiers are there?

The huge networks we found through trace-routing are known as Tier 1 networks.

Although there is no authority that defines tiers of networks participating in the Internet, the most common definition of a tier 1 network is one that can reach every other network on the Internet without purchasing IP transit or paying settlements.

By this definition, a tier 1 network is a transit-free network that peers with every other tier-1 network. But not all transit-free networks are tier 1 networks. It is possible to become transit-free by paying for peering or agreeing to settlements.

Common definitions of tier 2 and tier 3 networks:

  • Tier 2: A network that peers with some networks, but still purchases IP transit or pays settlements to reach at least some portion of the Internet.

  • Tier 3: A network that solely purchases transit from other networks to reach the Internet.

If you click through to Tier 1 networks from the Internet Backbone page you get to a list of the current Tier 1 networks:

  • AT&T from USA
  • Cogent Communications from USA
  • Centurylink (formerly Qwest and Savvis) from USA
  • Deutsche Telekom AG from Germany
  • GTT (formerly Tinet) from USA/Italy
  • Level 3 Communications from USA
  • Telecom Italia Sparkle from Italy
  • Telefonica Global Solutions from Spain
  • Verizon Business (formerly UUNET) from USA
  • TeliaSonera International Carrier from Sweden-Finlnd
  • NTT Communications from Japan
  • Tata Communications from India
  • Orange from France
  • XO Communications from USA
  • Zayo Group from USA

It is not known if AOL Transit Data Network (ATDN) is still a Tier 1 network.

Wait, what... What is Peering?

These networks connect to each other through a process known as 'peering'. Most traffic needs to go over at least two different top-tier networks in order to reach its destination, and the networks are bridged with peering arrangements. The way this usually works is that each party to the agreement will commit to routing x amount of traffic for the other party on their network, and vice versa. There is usually no money exchanged in these arrangements, unless one side is sending or receiving a lot more data than the other sides.

Large companies can also arrange their own peering relationships. For example Netflix has arranged its own peering and network infrastructure directly with multiple Tier 1 networks so that its traffic is both cheaper and closer to end users on each of the popular US broadband ISPs.

See this Wikipedia page on Peering.

There's a lot more to read at those pages; this answer gives a general idea, discovering all the details is left as an exercise to the reader. You can ask questions about this subject in the comments below...

  • 54
    Perhaps this would be suitable as a blog post as well? – Ivo Flipse Mar 11 '12 at 9:51
  • 10
    Because keeping it short and simple was never an option :D Excellent answer. +1 – Doomsknight Mar 11 '12 at 15:21
  • 2
    Perhaps it's useful to mention Internet Exchange Points ? You do state that many Tier-2 providers have peering arrangements, but don't mention how they physically do that. – MSalters Mar 13 '12 at 15:16
  • @MSalters: I'll try to look into that later and see what I can do. Please note that I didn't write the Peering part. nikcub was generous enough to add it to the answer... :) – Tom Wijsman Mar 13 '12 at 15:26
  • 2
    In summary, for the US: The phone company. – Joshua Drake Mar 13 '12 at 17:42

Basically as Tom described. To put his words into a generality, there are several internet backbones that connect with each other. ISPs work with these internet backbones and then sell bandwidth to their users. No one really owns the internet as it comprises millions of servers everywhere. However, there are central hubs (backbones) that can cause huge interruptions to several people's internet connectivity to some websites and servers. Basically, if a backbone in Colorado goes down, it could prevent East Coast US users to lost connectivity to servers on the West Coast. However, they would still have access to their own East Coast servers (assuming that the traffic isn't routed through Colorado in this example).

If you're wondering why a webpage may not be loading and there are other people complaining about the same things in your area then I would look at the internet health reports.

I just took this snapshot. It looks like there is high latency between Cogent and AT&T which could affect some users from accessing some sites or delay the responses.

enter image description here

SouthPark picture thrown in as related topic fun.

enter image description here

  • 47
    It was from the episode, "Over Lodging", where the internet went down. No one had internet access and no one could find the solution. The image above was the "backbone" of the internet that was causing the problem. South Park then made a mockery of it by making the backbone look like a Linksys WRT54G Router. One of the boys unplugged it and then plugged it back in. Internet was restored. – kobaltz Mar 11 '12 at 15:02

The Internet is a collection of networks. Most networks have to pay something to send or receive data to other networks. This includes the network in your home, even if the "network" only has 1 computer. Tier 1 networks do not have to pay to send or receive data with each other; the fact that they are not paying for privilege of sending/receiving traffic is the main thing that (unofficially) defines what is a Tier 1 network.

Note that the legal agreements that these Tier 1 players have with each other usually specify that any network they allow to send/receive data with them for free has to agree not to send too much more than they receive.

ISPs and other large networks interconnect with each other. There are two main types of interconnections, "peering" and "transit" (and a few variations between these two).

"peering" interconnections are usually but not always "settlement free" (neither network pays the other for the peering although there does need to be some arrangement to pay for the physical connection) and allow the two networks and their customers to communicate with each other. Peering is not transitive, if A peer with B and B peers with C then A doesn't get a route to C.

Peering may happen either on a direct link between the two networks or via exchange points. Exchange points provide an interconnect (nearly always an ethernet network) between many providers making peering viable between providers who will only exchange a small ammount of traffic. If two providers are exchanging a lot of traffic (sustained gigabits per second nowadays) then it's usually more economical to put in a direct link.

On a "transit" interconnection there is a provider-customer relationship. The transit provider provides (for a fee) the transit customer with connectivity to the entire internet. A customer may use multiple transit providers for redundancy or to provide shorter routes.

At the top of the tree there are a handful of big providers known as "tier 1" networks. These networks don't buy transit from anyone, instead they all peer with each other.

protected by nhinkle Mar 13 '12 at 20:48

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.