A/B Testing

Noun · Development

Definitions

  1. A/B Testing is an experimental method where two variants of a product, webpage, or feature are shown to different segments of users simultaneously to determine which performs better against a defined metric. Variant A typically represents the current version (control), while Variant B introduces a specific change (treatment). Statistical analysis determines whether observed differences in conversion rates, click-through rates, or other KPIs are significant rather than due to chance. A/B testing is foundational to data-driven product development and is used extensively by companies like Google, Netflix, and Amazon to optimize user experiences. Proper implementation requires sufficient sample size, random assignment, and a single variable change per test. More complex variations include multivariate testing (multiple changes) and multi-armed bandit approaches that dynamically allocate traffic to better-performing variants.

    In plain English: Showing two different versions of something to different users to see which one works better.

    Example: "We A/B tested the checkout button color and the green variant lifted conversions by 12%."

Etymology

1920s
Ronald Fisher's work on randomized experiments at Rothamsted provides the statistical foundation.
2000
Google pioneers large-scale A/B testing on search results, validating even tiny UI changes with data.
2007
Tools like Optimizely and Google Website Optimizer make A/B testing accessible to marketers and designers.
2010s
A/B testing becomes a core product management discipline. Netflix, Amazon, and others run thousands of concurrent experiments.

Origin Story

The Direct Mail Trick That Conquered the Internet

A/B testing, the practice of comparing two variants to see which performs better, has roots stretching back to the early 1900s when direct mail marketers would send different versions of a letter to random subsets of their mailing lists and measure response rates. The statistical foundations were laid by Ronald Fisher in the 1920s and 1930s through his work on randomized controlled experiments at the Rothamsted Experimental Station in England. The leap to the digital world happened gradually. In 2000, Google engineer Dan Siroker (who later co-founded Optimizely) helped run some of the first large-scale web A/B tests. Google famously tested 41 shades of blue for its link color in 2009, letting user click data determine the winner. The term became standard in web development during the mid-2000s as tools like Google Website Optimizer (launched 2007) made it easy for anyone to run experiments. Today, large tech companies run thousands of A/B tests simultaneously. Netflix tests everything from thumbnail images to recommendation algorithms, and Amazon has said that every feature starts as an A/B test. The methodology has become so central to product development that the ability to design and interpret A/B tests is now a core skill in product management and data science.

Context: Direct mail marketing / statistical experimentation, early 1900s

Fun fact: Google's famous '41 shades of blue' test reportedly generated an extra $200 million in annual ad revenue by finding the optimal link color.

Related Terms