p-value

Definitions

1. The probability of observing a result at least as extreme as the one measured, assuming the null hypothesis is true. A p-value of 0.03 means that if there were genuinely no effect, data this extreme would arise about three percent of the time. It is routinely and consequentially misread: it is not the probability that the hypothesis is false, not the probability the result arose by chance, and not a measure of effect size. A tiny p-value from a huge sample can accompany an effect too small to matter. The conventional 0.05 threshold is arbitrary and, combined with publication incentives, drives p-hacking, in which analyses are varied until something crosses the line. The American Statistical Association issued a formal statement in 2016 cautioning against exactly these misuses, and many fields now require effect sizes and confidence intervals alongside.

   In plain English: A number that tells you how likely your result happened by pure luck. A very small p-value means your result is probably real, not just a coincidence.

   Example: The marketing team got excited about a 12% lift in click-through rate, but the p-value was 0.34 — not even close to significant with their sample size.

Related Terms

• Latency
• Throughput
• tail latency
• A/B testing
• correlation
• regression
• Cross-Validation
• Gaussian Mixture Model