1

I am trying to understand and draw network path from traceroute output. An example output looks like below:

Suppose I want to draw the path between 3rd and 4th hop. Is it right to assume that there will be ordered mapping between ips across different levels?

Meaning the packet will go from 1st ip (et-0-0-2.mpr01.mtlcx03.fibrenoire.ca) of hop 3 to 1st ip of hop 4(et-0-0-4.cre01.mtl1981.fibrenoire.ca) and the second packet will go from 2nd ip (et-0-0-3.mpr01.mtlcx03.fibrenoire.ca) of hop3 to 2nd ip (et-0-0-4.cre01.mtlsunl.fibrenoire.ca) of hop 4? and this will continue for next hops ?

Or a second example between level 7 and level 8, is the packet guaranteed to go from 52.94.82.32 to 52.94.81.193 only and another packet from 52.94.81.134 to 52.94.81.191?

| #  | Hostname                              | Loss% |  Snt | Last |  Avg | Best |  Wrst |
+----+---------------------------------------+-------+------+------+------+------+-------+
| 1  | 192.168.1.1                           |  72.8 | 2220 |  3.2 |  2.0 |  0.6 |  34.4 |
| 2  | CUST01-C1.asr02.mtl1080.fibrenoire.ca |   0.0 | 2220 |  2.0 |  3.6 |  1.6 |  51.0 |
| 3  | et-0-0-2.mpr01.mtlcx03.fibrenoire.ca  |   0.0 | 2220 |  1.4 |  4.5 |  1.0 |  66.7 |
|    | et-0-0-3.mpr01.mtlcx03.fibrenoire.ca  |       |      |      |      |      |       |
| 4  | et-0-0-4.cre01.mtl1981.fibrenoire.ca  |   0.0 | 2220 |  1.2 |  5.6 |  1.1 |  84.9 |
|    | et-0-0-4.cre01.mtlsunl.fibrenoire.ca  |       |      |      |      |      |       |
| 5  | et-0-0-2.mpr02.mtlcx03.fibrenoire.ca  |   0.0 | 2220 |  1.4 |  5.6 |  1.2 | 135.0 |
|    | et-0-0-3.mpr02.mtlcx03.fibrenoire.ca  |       |      |      |      |      |       |
| 6  | 52.95.219.62                          |   0.0 | 2220 |  3.3 |  4.3 |  1.1 |  78.2 |
| 7  | 52.94.82.32                           |   0.0 | 2220 |  2.7 |  4.4 |  1.8 |  67.4 |
|    | 52.94.81.134                          |       |      |      |      |      |       |
| 8  | 52.94.81.193                          |   0.0 | 2220 |  7.8 |  3.3 |  1.4 |  56.3 |
|    | 52.94.81.191                          |       |      |      |      |      |       |
| 9  | 52.94.82.69                           |   0.0 | 2220 | 28.2 | 26.4 | 22.4 |  60.5 |
|    | 54.239.44.18                          |       |      |      |      |      |       |
| 10 | 150.222.242.150                       |  48.6 | 2220 | 28.9 | 26.4 | 22.3 |  50.5 |
|    | 150.222.242.118                       |       |      |      |      |      |       |
| 11 | 150.222.242.150                       |  89.1 | 2220 | 28.9 | 26.1 | 22.4 |  37.7 |
|    | 150.222.242.152                       |       |      |      |      |      |       |
2
  • 1
    well, depending on implementation, traceroute works by using TTL and increasing it by one for each successive probe, so that each probed hop responds with a TTL EXCEEDED error. so yes, each successive hop will be the next node along the path. am I perhaps misunderstanding your question? its kind of hard to tell what you are actually asking. Commented Apr 5, 2022 at 7:32
  • Thanks @FrankThomas, I added a second example. Basically I want to know is the path guaranteed? Commented Apr 5, 2022 at 8:03

1 Answer 1

1

is the packet guaranteed to go from 52.94.82.32 to 52.94.81.193 only and another packet from 52.94.81.134 to 52.94.81.191

tl;dr: no, that is not guaranteed.

Whether those addresses are shown in any specific order actually depends on which traceroute program you use. Each does its own thing. (They might even be using different kinds of probes for tracing – e.g. Windows tracert uses ICMP probes exclusively, Linux traceroute prefers UDP unless told otherwise, and in your case they will show slightly different paths.)

But in general, what you're looking at is the result of ECMP (equal-cost multipath) routes, where e.g. the router at hop 2 has a 50%:50% choice between two gateways for hop 3. (Then hop 3 might also use ECMP and make another, independent choice between two gateways for hop 4... so it might be not merely 2 paths, but many possible combinations.)

This choice is made based on packet headers, trying to ensure that all packets belonging to the same "connection" will travel the same path – which usually means the coin flip is based on the source/destination IP addresses and TCP/UDP ports. (For example, with everything else being the same, packets to odd ports could take path A while even ports take path B.)

(Note that each hop makes completely independent routing decisions – gateway A could have a 2-way ECMP route based on IP+port, but at the same time gateway C could have a 4-way route based on IP only, etc.)

If your traceroute program uses UDP probes and sends each probe to a different UDP destination port, then it's possible that the results will be useless – sometimes you might see both paths intermingled (like in your example), sometimes you might not notice that e.g. hop 3 is from path A, hop 4 is from path B, hop 5 is from path A again.

So in output like yours, I would not make assumptions that the order of addresses for hop 4 has any relationship with the order for hop 5.

What you should do is tell your traceroute tool to use fixed UDP source/destination ports, and run the trace several times while manually altering the port between each trace. That way all probes for a single trace will take a single path. For example, if you're using mtr (whose default is actually ICMP), try:

  • mtr <host> – this uses ICMP and should always show one path;
  • mtr --udp <host> – this uses UDP with varying ports and will show many paths at once, with no possible way to distinguish them;
  • mtr --udp -L 33000 -P 33001 <host> – this uses UDP with local port 33000, remote port 33001, and will show you one path only (let's call it path A);
  • mtr --udp -L 33000 -P 33002 <host> – same as above, but adjusting either one of the port numbers may show you a different path (path B).

Here's an example of two independent instances of ECMP within a single trace – first you have hop 4 choose between gateways in different countries (which converge in uk-hex; the two addresses of hop 6 are most likely to be just different interfaces of the same router), then you have hop 7 choose between eight different routers in the end.

$ mtr --udp -c 100 --report-wide 212.71.255.###
                                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- _gateway (10.128.0.1)                           0.0%   100    0.5   0.4   0.2   0.7   0.1
  2.|-- uxxx.lxxxxx.lx (193.219.###.###)                0.0%   100    0.8   0.7   0.4   1.1   0.1
  3.|-- ktu-itd-gw (193.219.153.182)                    0.0%   100    6.0   6.1   5.4  22.2   1.9
  4.|-- de-hmb.nordu.net (109.105.98.126)               0.0%   100   31.8  31.2  30.5  34.6   0.7
  5.|-- de-ffm.nordu.net (109.105.97.105)               0.0%   100   40.3  39.0  37.0  63.8   2.8
        nl-ams.nordu.net (109.105.97.80)              
  6.|-- uk-hex.nordu.net (109.105.97.125)               0.0%   100   45.9  46.4  45.4  54.5   1.6
        uk-hex.nordu.net (109.105.97.78)              
  7.|-- 5.57.81.142                                     0.0%   100   47.2  49.0  46.1  78.5   4.0
  8.|-- if-3-3.router1-lon1.linode.com (109.74.207.19)  0.0%   100   50.5  48.8  46.3  51.6   1.9
        if-3-23.router1-lon1.linode.com (109.74.207.23)
        if-3-19.router2-lon1.linode.com (109.74.207.3)
        if-3-23.router2-lon1.linode.com (109.74.207.25)
        if-3-3.router2-lon1.linode.com (109.74.207.21)
        if-3-22.router2-lon1.linode.com (109.74.207.15)
        if-3-19.router1-lon1.linode.com (109.74.207.1)
        if-3-17.router1-lon1.linode.com (109.74.207.11)
  9.|-- wxxxx.nxxxxxxxx.xx (212.71.255.###)             0.0%   100   46.2  48.3  46.0  51.3   1.9

If you looked at this using Wireshark/tcpdump, you'd see mtr sending each probe to a different UDP port – 33000, then 33001, then 33002, etc. – which is how it ends up collecting all possible combinations over time.

But if I specify fixed source/destination ports for UDP probes (or if I don't use --udp in the first place, removing ports from the equation), then I can isolate a single path:

$ mtr --udp -L 33000 -P 33000 -c 10 --report-wide 212.71.255.###
                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- _gateway (10.128.0.1)                            0.0%    10    0.4   0.4   0.4   0.6   0.1
  2.|-- uxxx.lxxxxx.lt (193.219.###.###)                 0.0%    10    0.7   0.7   0.6   0.8   0.1
  3.|-- ktu-itd-gw (193.219.153.182)                     0.0%    10    6.4   5.9   5.7   6.5   0.3
  4.|-- de-hmb.nordu.net (109.105.98.126)                0.0%    10   30.8  30.9  30.8  31.4   0.2
  5.|-- de-ffm.nordu.net (109.105.97.105)                0.0%    10   39.3  40.4  39.3  49.3   3.1
  6.|-- uk-hex.nordu.net (109.105.97.78)                 0.0%    10   46.0  47.4  45.8  51.4   1.9
  7.|-- 5.57.81.142                                      0.0%    10   51.7  50.5  50.1  51.7   0.4
  8.|-- if-3-17.router1-lon1.linode.com (109.74.207.11)  0.0%    10   50.4  50.6  50.3  51.6   0.4
  9.|-- wxxxx.nxxxxxxxx.xx (212.71.255.###)              0.0%    10   50.3  50.3  50.2  50.5   0.1

And changing one of the port numbers gives me a different path:

$ mtr --udp -L 33000 -P 33002 -c 10 --report-wide 212.71.255.###
                                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- _gateway (10.128.0.1)                           0.0%    10    0.6   0.4   0.4   0.6   0.1
  2.|-- uxxx.lxxxxx.lx (193.219.181.254)                0.0%    10    0.7   0.6   0.5   0.8   0.1
  3.|-- ktu-itd-gw (193.219.153.182)                    0.0%    10    5.8   6.0   5.8   6.7   0.4
  4.|-- de-hmb.nordu.net (109.105.98.126)               0.0%    10   30.8  30.8  30.7  31.0   0.1
  5.|-- nl-ams.nordu.net (109.105.97.80)                0.0%    10   37.4  39.5  37.2  56.1   5.9
  6.|-- uk-hex.nordu.net (109.105.97.125)               0.0%    10   45.9  46.0  45.7  47.4   0.5
  7.|-- 5.57.81.142                                     0.0%    10   46.6  48.1  46.5  57.5   3.6
  8.|-- if-3-3.router1-lon1.linode.com (109.74.207.19)  0.0%    10   47.1  46.9  46.4  48.2   0.5
  9.|-- wxxxx.nxxxxxxxx.xx (212.71.255.###)             0.0%    10   46.6  46.5  46.3  46.6   0.1

Now returning to your original question, notice how path A went through gateways → 97.105 → 97.78 →, while path B went → 97.80 → 97.125 →, and this clearly does not correspond to the order of the hops in the initial multi-path trace (which shows A then B in position 5, but the opposite in position 6).

Since 97.78 and 97.125 are actually the same gateway (the looking glass confirms this), it would be unlikely for a packet to travel from 97.105 to 97.125 because that interface is only connected to 97.80. From that we can conclude that the order of addresses within a single 'hop' in mtr's output is completely meaningless.

4
  • Really thanks for such a detailed example. 1 doubt when you suggest not to make assumptions on order of addresses. So through traceroute my use case is not to predict future results. My use case is to plot the path that tarceroute followed in past. So in your example (first one), between hop 5 and 6, would it still be wrong to assume that a packet went from 109.105.97.105(hop5) to 109.105.97.125(hop6) and other packet from 109.105.97.80(hop5) to 109.105.97.78(hop6)? Commented Apr 5, 2022 at 8:11
  • This answer deserves a bounty. Unfortunately I do not have enough reputation but I really acknowledge the effort. Commented Apr 5, 2022 at 8:17
  • 1
    @tryingToLearn: Yes, I just updated the post about that. From what I was able to find out, there are separate point-to-point links going de-ffm→uk-hex and nl-ams→uk-hex, and each individual point-to-point link uses a distinct /30 or perhaps /31 subnet (so the same router has many addresses). So if uk-hex is responding to a packet received via its "de-ffm" interface, it should always reply from the IP address assigned to that interface, and likewise if it received the probe from nl-ams it'll respond using its "nl-ams"-facing address, therefore the 105→125 and 80→78 combinations are ruled out. Commented Apr 5, 2022 at 8:17
  • 1
    @tryingToLearn: Though, I am actually not 100% sure about whether my own last comment is correct. One more complication that I just remembered is that paths aren't always symmetric – even if probes take path A, replies might actually take path B, so the hops visible in a trace actually could be using a source IP address belonging to a different interface. Upon further investigation, it actually seems like when the path is de-hmb→de-ffm→uk-hex the latter will respond directly through uk-hex→de-hmb (or at least that's what the IP addresses tell me). Commented Apr 5, 2022 at 8:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .