Read latency beyond the average.
A service can show a modest average while a small group of requests takes much longer. The distribution determines how many requests that average conceals.
Follow the slow group
Consider a constructed sample of 1,000 completed requests. In the first version, every request takes 100 milliseconds. In the second, 985 still take 100 milliseconds and 15 take 1,200 milliseconds. The mean rises from 100 to 116.5 milliseconds. Those 15 requests each take twelve times as long.
Using the nearest-rank method, the 95th percentile is the 950th value in the ordered sample and the 99th percentile is the 990th. In the second sample, the 950th request is still in the 100-millisecond group. The 990th is in the slower group, so the 99th percentile is 1,200 milliseconds.
Illustrative request durations
- Mean
- 100 ms
- 95th percentile
- 100 ms
- 99th percentile
- 100 ms
Every request in this sample takes 100 ms.
Keep the request group visible
An aggregate chart may combine search, page loads and exports. An export that normally takes several seconds can obscure a change in a short interactive request. Segment by the request type needed for the investigation, while keeping enough observations to make the comparison useful.
Keep the request count beside the percentile. In a small sample, one request can move the selected value sharply. The distribution in this example is exact; a monitoring system may estimate its percentiles from buckets or a sampled stream.
Treat timeouts separately
A measurement of completed requests may omit requests that timed out or were abandoned. The latency view needs an accompanying count of errors and timeouts. Otherwise, a service can appear faster after its slowest requests stop completing.
Check how the monitoring tool calculates percentiles and aggregates them across hosts. Averaging several host-level 99th percentiles does not produce the 99th percentile of their combined requests. The combined distribution or a suitable histogram is needed for that calculation.
Choose the interval before comparing two releases
A quiet five-minute interval and a busy hour may include different endpoints, payload sizes and dependency pressure. Keep the collection rule visible when comparing them. A change in the mix can move the percentile even if the behavior of each endpoint stays the same.
For a release comparison, inspect a relevant request group using the same measurement definition. Report the count with the percentile. A p99 derived from a small number of requests has a different interpretation from one supported by a large sample.
Use the chart to narrow the investigation
A slower tail identifies a group to inspect. Traces can show which dependency call or application segment took time; runtime measurements can show whether resource saturation occurred in the same interval. Compare the same endpoint and measurement method when assessing a change.
Service monitoring