A/B testing
/ay-bee TEST-ing/ · noun · Development · Origin: 2000
Definitions
A controlled experiment where two variants (A and B) of a webpage, feature, or algorithm are shown to randomly split user groups to determine which performs better on a target metric. Statistical significance is used to ensure observed differences are not due to chance. The practice extends to multivariate testing and bandit algorithms.
In plain English: Showing two different versions of something to separate groups of users to see which one works better — like testing two headlines to see which gets more clicks.
Example: The A/B test showed the green checkout button increased conversions by 4.2% with a p-value of 0.003, so they shipped it to all users.
Etymology
- 1920s
- Statistician Ronald Fisher develops randomized controlled experiments, the intellectual foundation of A/B testing.
- 2000
- Google begins systematic A/B testing of search results, proving the method's power at web scale.
- 2007
- Optimizely and similar tools democratize A/B testing, making it accessible to non-engineers.
- 2010s
- A/B testing becomes standard practice in product development. 'Data-driven' decision-making takes hold.
- 2020s
- Multi-armed bandit and Bayesian methods evolve the practice beyond simple two-variant tests.
Origin Story
The controlled experiment that settled arguments with data
**A/B testing** in software traces to Google engineer Dan Siroker, who ran one of the first large-scale web A/B tests in 2000 when Google tested 41 shades of blue for ad links. But the method itself is much older -- it's just **randomized controlled trials** applied to software.
Ronald Fisher formalized experimental design in the 1920s-1930s for agriculture. The web made it cheap and fast: instead of growing crops for months, you could test a button color on millions of users in hours. Optimizely (co-founded by Siroker) democratized A/B testing in 2010.
A/B testing's dark side emerged as companies optimized relentlessly for engagement metrics. Facebook and YouTube A/B-tested their way to addictive algorithms. The tests always found a 'winner,' but optimizing for clicks isn't the same as optimizing for user well-being.
Coined by: Ronald Fisher (experimental design, 1920s), Google (web A/B testing, 2000)
Context: Statistical roots 1920s; web application 2000s
Fun fact: Google's '41 shades of blue' test (choosing between two blues for toolbar links) was controversial internally. Designer Doug Bowman quit Google over it, writing that 'data eventually becomes a crutch for every decision.' The shade that won reportedly earned $200 million in additional ad revenue.