I normally have `mtr 1.1`¹ running in the background, in the third display mode, which is a 2D histogram—time in the x axis, hops in the y axis, and ASCII character/colour for ping time. When problems occur, this tends to let you easily see the nature of the problem—total loss, elevated packet loss, elevated response times; and to see the location of the problem—local network, local ISP, public internet. There are definitely occasions for loss%, sent, last/average/best/worst/stddev ping and such things as are found in the first display mode, but most of the time I find the histogram view most useful as the starting point.
You can make mtr start in this view with --displaymode=2 (direct command line arguments, `mtr --displaymode=2 …`; or shell alias, `alias mtr="mtr --displaymode=2"`; or set environment variable MTR_OPTIONS=--displaymode=2).
MTR is a useful tool but it is a somewhat common source of illusory issues since it generates so many icmp time exceeded packets that routers stop replying to other folks running traces. It's important, as others said, to understand that these aren't testing the data path of a network but instead the control plane path.
What tools exist for people to test the data path of network reliably? In my experience MTR has worked well enough to approximate network routing issues. But it has always felt like a blunt tool given it can't do anything about hops with firewalls.
MTR has long been one of the first little tools that I install on workstations.
sudo apt install mtr-tiny
I also have a hotkey to pop it up in a window, pinging to some host that'll always be somewhere on the other side of any ISP from me. Whenever I suddenly suspect a networking problem from my laptop, I hit the hotkey as the first troubleshooting step. MTR starts to narrow down a few different problems very quickly.
> Is it indicating that the router is faster at forwarding packets than responding to ping requests?
Exactly this. In most “real” routers, forwarding (usually) happens in the “data plane”. It’s handled by an ASIC that has a routing table accessible to it in RAM. A packet comes in on an interface, a routing decision is made, and it goes out another interface - all of this happens with dedicated hardware. Pings (ICMP Echo requests), however, get forwarded by this ASIC to a local CPU, where they are handled by software (in the “control plane”).
You’re really seeing different response times from the two control planes - one may be more loaded or less powerful than another, regardless of the capacity of their data planes.
Traceroute doesn't use ping requests except with the old Windows binary. Usually it uses "Time-to-live (TTL) exceeded in transit" messages.
Beyond that technicality, your guess is often right... Routers will frequently prioritize forwarding packets over sending the TTL exceeded packets tools like MTR use to measure response times.
Most likely, it's as you described, router N forwards packets much faster than it generates icmp ttl exceeded, and router N+1 is nearby and generates icmp faster.
However, it could also be the case that the routing back to you is significantly different, so you can have a much longer path to you from router N than router N+1.
This is more likely to happen on routes that cross oceans. Say you're tracing from the US to Brazil. If router N and N+1 are both in Brazil, but N sends return packets through Europe and N+1 sends through Florida, N+1 returns will arrive significantly sooner.
The results aren't misleading, shockingly large numbers of "computer professionals" have no idea how networks work, but that's because they can't use the tools rather than the tools being misleading
chrismorgan|1 year ago
You can make mtr start in this view with --displaymode=2 (direct command line arguments, `mtr --displaymode=2 …`; or shell alias, `alias mtr="mtr --displaymode=2"`; or set environment variable MTR_OPTIONS=--displaymode=2).
Screenshot of this mode: https://temp.chrismorgan.info/2025-02-06-hn-42924182-mtr-dis...
—⁂—
¹ 1.1 = 1.0.0.1 = Cloudflare public DNS, a convenient nearby public internet endpoint.
jlmcguire|1 year ago
cootsnuck|1 year ago
neilv|1 year ago
eudhxhdhsb32|1 year ago
One thing I've not understood is why will some hops have consistently lower ping times than hops farther down the chain in the same trace?
Is it indicating that the router is faster at forwarding packets than responding to ping requests?
p_ing|1 year ago
https://archive.nanog.org/sites/default/files/traceroute-201...
wrigby|1 year ago
Exactly this. In most “real” routers, forwarding (usually) happens in the “data plane”. It’s handled by an ASIC that has a routing table accessible to it in RAM. A packet comes in on an interface, a routing decision is made, and it goes out another interface - all of this happens with dedicated hardware. Pings (ICMP Echo requests), however, get forwarded by this ASIC to a local CPU, where they are handled by software (in the “control plane”).
You’re really seeing different response times from the two control planes - one may be more loaded or less powerful than another, regardless of the capacity of their data planes.
oxygen_crisis|1 year ago
Beyond that technicality, your guess is often right... Routers will frequently prioritize forwarding packets over sending the TTL exceeded packets tools like MTR use to measure response times.
toast0|1 year ago
However, it could also be the case that the routing back to you is significantly different, so you can have a much longer path to you from router N than router N+1.
This is more likely to happen on routes that cross oceans. Say you're tracing from the US to Brazil. If router N and N+1 are both in Brazil, but N sends return packets through Europe and N+1 sends through Florida, N+1 returns will arrive significantly sooner.
rixed|1 year ago
I believe most of the time this is the reason indeed. Answering an ICMP error to a TTL expiration or to an echo request is very low priority.
This latency in error message generation may even be a better signal of the router load than the latency of the actualy trip through it.
commandersaki|1 year ago
ta1243|1 year ago
otterz|1 year ago
walrus01|1 year ago
https://www.youtube.com/watch?v=L0RUI5kHzEQ
klysm|1 year ago