Average Values Alone Hide Data Distribution: Importance of Visualization Techniques

Software engineers risk making erroneous judgments when relying solely on the mean to verify performance improvements. According to a report from fzakaria.com via domenkozar on Lobsters, a case was introduced where an engineer evaluating performance improvements in the linker “lld” faced the issue that improvements seen in benchmarks were not reflected in production dashboards.

Misleading Judgments from Averages

A colleague of the author encountered similar difficulties when evaluating improvements in build times. Build times are influenced by numerous variables such as cold cache, incremental builds, local environments, and remote environments, and fluctuate greatly depending on system state and workload. She solved this problem by adopting visualization using the cumulative distribution function (CDF).

From this experience, the author concludes that a single statistical value or a single graph cannot capture the full picture of the data. Verification using synthetic datasets showed that the same data can lead to completely different stories depending on the visualization method used.

When Mean and Median Contradict

Consider a case where a new cache hierarchy was introduced to improve web service latency. Rolled out gradually over one week, after full deployment the average latency rose from 112 milliseconds to 122 milliseconds. At first glance, this appears to be a regression, reported as a SEV (Severity Incident), prompting consideration of rollback and postmortem creation.

However, examining percentiles in detail reveals a completely different situation. The p50 (median) improved 46%, from 99 milliseconds to 54 milliseconds, while the p95 worsened 103% from 224 ms to 454 ms, and the p99 worsened 119% from 309 ms to 678 ms. Despite being calculated from the same data, the mean and median point in entirely opposite directions.

Interpreting Distribution Shapes

Relying on single statistics like the mean or median makes it easy to engage in “cherry picking,” selecting convenient numbers to support one’s claims. To understand the true nature of data, visualizing the distribution shape is essential.

Using density plots to compare latency distributions before and after the change reveals that the pre-change distribution is broad, while the post-change distribution shows a sharp concentration in the center but a longer tail on the right side. This indicates that latency improved for the majority of requests but worsened significantly for a small minority.

Benefits of CDF Visualization

Using the cumulative distribution function (CDF) allows intuitive understanding of what percentage of requests are below each latency value. For example, it is immediately clear how the proportion of requests under 120 ms changed, and how the 90th percentile latency value changed.

CDFs are especially suitable for observing tail behavior, accurately capturing situations where a small number of outliers disproportionately raise the mean. This visualization can quantitatively show a situation where “the average went up, but for most users it’s an improvement.”

Practical Implications

In the field of performance evaluation, the problem of benchmark results not matching production metrics occurs frequently. The author points out that one cause is the evaluation method relying on a single statistic like the mean.

By evaluating using multiple percentiles (p50, p95, p99) in combination with CDFs, the impact of system changes can be understood from multiple angles. Additionally, overlaying the before and after distributions allows answers to qualitative questions such as “for whom is this an improvement?” and “under what conditions does it worsen?”

Editorial Opinion

Performance evaluation that relies solely on the mean carries the risk of misinterpreting the essence of system changes. Particularly in web service latency measurements, behavior in the tail directly affects user experience, making multi-faceted analysis using percentiles and CDFs indispensable. In the short term, engineering teams are likely to advance efforts to redesign dashboards and review incident response procedures.

In the long term, practices that incorporate CDF-based automatic regression detection into CI/CD pipelines may become widespread. Major development platforms such as GitHub and GitLab are also expected to standardize CDF and percentile distributions as default visualization methods for performance measurement.

To correctly evaluate the impact of system changes, fostering a culture of “viewing data holistically” is necessary. The perspectives introduced in this article will likely ripple through the entire software engineering quality management process.

References

• “The mean means nothing”, by fzakaria.com via domenkozar — Lobsters, 2026-07-28T18:53:28.000Z (ARR)
• Source URL: https://fzakaria.com/2026/07/27/the-mean-means-nothing