Introduction
In the intricate world of web services, data is the lifeblood. Yet, relying solely on the mean to assess performance can be misleading. The case of Farid Zakaria, who attempted to optimize lld performance at work, highlights this issue. Production dashboards didn't reflect the improvements visible in benchmarks. This discrepancy prompts a reevaluation of our relationship with the mean.
The Trap of the Mean
The mean is often used to simplify complex data sets. However, it only tells part of the story. For instance, an average latency increase from 112 ms to 122 ms might seem concerning. Yet, this single figure can obscure significant variations in the data.
Consider a concrete example: a web service deploying a new caching tier to reduce request latency. The mean indicates an increase in latency, but a more detailed analysis reveals different stories across percentiles:
- Median (p50): Median latency drops from 99 ms to 54 ms, a 46% improvement.
- P95: Latency for the worst 5% of requests nearly doubles, from 224 ms to 454 ms.
- P99: An even sharper increase with latency jumping from 309 ms to 678 ms.
Looking Beyond the Numbers
Data visualizations like cumulative distribution functions (CDFs) provide a better understanding of the overall data distribution. A CDF shows the probability that a variable is less than or equal to a certain value, offering a more comprehensive perspective.
The Importance of Percentiles
Percentiles such as p50, p95, and p99 are essential for understanding system performance in production. For example, while the mean increased, a significant drop in the median indicates that the majority of requests are faster. Conversely, the increases in p95 and p99 reveal that the worst-case scenarios have worsened.
Real-World Examples
Consider large tech companies like Amazon or Google, where performance optimization is critical. These companies often use percentiles to assess the impact of changes on their systems. Google, for instance, doesn't just look at the mean but regularly examines p95 and p99 to ensure performance remains optimal even in the worst cases.
Conclusion
The mean should not be the sole performance indicator. Tools like CDFs and percentile analysis provide a more nuanced and accurate view. As a decision-maker or engineer, adopting a comprehensive approach is crucial to avoid misinterpretations of data.
Let's discuss your project in 15 minutes.