mtr

mtr combines the functionality of the 'traceroute' and 'ping' programs in a single network diagnostic tool. 

Matt's Traceoute is an application you can use to do a repeated traceroute in real time; it dynamically shows the round-trip time to reach each hop along the traceroute path. The constant updates enable you not only to visually determine which hops are slow, but also to determine when they appear to be slow. It is a good tool to use whenever you suspect there is some intermittent network congestion. 

Each jump is consider a hop. 

The next four columns Last, Avg, Best, and Wrst are all measurements of latency in milliseconds (e.g. ms). Last is the latency of the last packet sent, Avg is average latency of all packets, while Best and Wrst display the best (shortest) and worst (longest) round trip time for a packet to this host. 

The final column, StDev, provides the standard deviation of the latencies to each host. The higher the standard deviation, the greater the difference is between measurements of latency. Standard deviation allows you to assess if the mean (average) provided represents the true center of the data set, or has been skewed by some sort of phenomena or measurement error. 

In most cases, the average (Avg) column should be the focus of your attention.


Verifying Packet Loss

When analyzing MTR output, you are looking for two things: loss and latency. First, let's talk about loss. If you see a percentage of loss at any particular hop, that may be an indication that there is a problem with that particular router. However, it is common practice among some service providers to rate limit the ICMP traffic that MTR uses. This can give the illusion of packet loss when there is in fact no loss. To determine if the loss you're seeing is real or due to rate limiting, take a look at the subsequent hop. If that hop shows a loss of 0.0%, then you can be pretty sure that you're seeing ICMP rate limiting and not actual loss.


INFO : http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch04_:_Simple_Network_Troubleshooting#Using_MTR_To_Detect_Network_Congestion

       http://library.linode.com/linux-tools/mtr