Demystifying the p-value: The Essence of Statistical Significance

Based on an article reported by Kyle Hewitt, a neuroscientist and associate lecturer specializing in neuropsychopharmacology at the University of Wollongong, published in The Conversation - Technology.

Anyone who has read a scientific paper has almost certainly encountered the term “p-value.” If you’ve ever been confused about how to interpret it, you are not alone. The reality is that even many practicing scientists do not fully understand what the p-value represents.

The p-value is a number that measures the “statistical significance” of an experiment and often serves as the deciding factor in whether the experiment has truly discovered something. Whether researchers can claim that a drug had an effect or that one group differed from another depends on the p-value. Careers, publications, and even our understanding of the world hinge on this single number.

The p-value is a number measuring the “statistical significance” of an experiment, and it’s often treated as the adjudicator of whether the experiment really found something or not. It decides whether researchers get to say that a drug did something, or that one group differed from another. Careers, publications, and our understanding of the world hinge on p-values. So it’s worth understanding what these numbers actually mean, because it’s not what most people think.

What the p-value Indicates—and What It Does Not

To understand the p-value, let’s consider a concrete example: Suppose you want to determine whether adults in one country are taller on average than adults in another country. Since you cannot measure everyone, you take a sample of 50 people from each country. The average height in the first sample is 1.80 meters, while in the second it is 1.77 meters—a difference of 3 centimeters. On its own, this difference means little because random variation in sampling can naturally produce differences between any two data sets.

The question is whether this 3 cm difference is reliable evidence that there is a real average height difference between the two countries, or if it is merely the result of chance. To answer this, scientists conduct a statistical test and calculate the p-value. Let’s assume the test produces a p-value of 0.06.

The meaning of this p-value is as follows: If there were truly no difference between the two countries’ average heights, the probability of observing a difference as large as 3 cm—or larger—purely due to random sampling is approximately 6%.

It is crucial to note what the p-value does not mean. A p-value of 0.06 does not mean there is a 6% probability that the result is due to chance. Nor does it mean there is a 94% probability that there is a real difference between the two countries. Instead, the p-value tells us the probability of obtaining such a large difference if there were no real difference to begin with. Confusing these two interpretations is a very common mistake, even among well-informed individuals.

The Origin of the 0.05 Threshold

By convention, results are considered “statistically significant” if the p-value is below 0.05. In the height study example, because the p-value is 0.06, the difference would not be reported as statistically significant.

But where does the 0.05 threshold come from? The truth is, it does not have any rigorous justification. It was proposed as a rule of thumb by British statistician Ronald Fisher in the 1920s and gained widespread acceptance due to its convenience.

Setting the cut-off for statistical significance at 0.05 has a practical implication: If there is truly no effect, this threshold ensures that roughly 1 in 20 experiments will yield a “statistically significant” result purely by chance. This is the false positive rate we’ve collectively agreed to tolerate. However, this does not indicate how many published research findings are incorrect.

What is arbitrary is the choice of one in 20 (0.05) as opposed to one in 100 (0.01) or one in 10 (0.10). As a result, this threshold produces somewhat absurd outcomes, where a p-value of 0.049 is celebrated as a discovery, while a p-value of 0.051 is treated as though nothing happened.

What is arbitrary is one in 20 rather than one in 100 or one in ten, and it produces faintly absurd outcomes: if p = 0.049, it’s reported as a discovery; if p = 0.051 we act as though nothing happened.

”Not Significant” Does Not Mean “No Difference”

Returning to the height experiment: Looking at a p-value of 0.06, one might be tempted to conclude that there is no difference in height between the two countries. However, this interpretation is incorrect.

A result that is not statistically significant does not prove that there is no effect. It merely indicates that the available data do not provide strong enough evidence to conclude that an effect exists. This could be due to a small sample size or the genuine absence of an effect—but the p-value alone cannot distinguish between these possibilities.

This misunderstanding has caused significant problems in fields like AI and machine learning. For example, in comparing model performance, researchers sometimes claim “significant” results based on minor differences that happen to reach the threshold or dismiss promising approaches simply because they fail to achieve significance. Such practices have contributed to the reproducibility crisis in research.

To improve the reliability of scientific studies, researchers must move beyond over-reliance on statistical significance thresholds. Instead, they should also consider effect sizes, confidence intervals, and, most importantly, the practical significance of their findings.

Editorial Opinion

Misinterpretations of the p-value reflect a fundamental issue in statistical education. As fields like AI and data science rapidly expand, reliance on p-values for interpreting model evaluations or A/B test results remains strong.

In the short term (3–6 months), there could be stricter reporting practices for statistical significance in evaluation papers, such as those involving large language models. Some top-tier conferences are already accelerating efforts to require reporting not only p-values but also effect sizes and Bayes factors.

From a long-term perspective (1–3 years), a lack of statistical literacy risks undermining the overall quality of AI research. This is particularly concerning in critical domains like medical AI and autonomous driving, where statistical misinterpretations could have life-altering consequences. The research community must work toward establishing alternative evaluation criteria and adopting stricter distinctions between exploratory and confirmatory studies through preregistration.

As an editorial team, we pose this question: The 0.05 p-value threshold was reportedly a convenient figure proposed by Fisher in the 1920s while he “thought about it over a cigarette.” How seriously should we confront the reality that this arbitrary rule from a century ago still dictates the validity of “discoveries” in cutting-edge AI research? Are we using p-values with a full understanding of their limitations?

References

• “What is a p-value? An expert explains the most misunderstood number in science”, by Kyle Hewitt, Neuroscientist and Associate Lecturer specialising in neuropsychopharmacology, University of Wollongong — The Conversation - Technology, 2026-07-28T20:08:49.000Z (CC BY-ND)
• Source URL: https://theconversation.com/what-is-a-p-value-an-expert-explains-the-most-misunderstood-number-in-science-288491