CUPED

Gold definitionUpdated Apr 2, 2026

CUPED is a statistical method designed to enhance the precision of treatment effect estimates in randomized controlled trials, commonly known as A/B tests. The core mechanism involves using a pre-experiment covariate, typically a measurement of the outcome variable before the experiment begins, to statistically control for baseline differences among experimental units. This control is achieved by adjusting the observed post-experiment outcome for each unit by subtracting a fraction of their pre-experiment value, effectively 'controlling' for their initial state. This process substantially reduces the unexplained variance in the outcome, making A/B tests more sensitive to detect smaller changes, reducing the required sample size, and shortening experiment durations. It is widely adopted by major tech companies like Microsoft, Google, and Meta for evaluating product changes, feature rollouts, and algorithmic updates across various domains, including e-commerce, social media, and conversational AI, where it enables highly sensitive A/B testing for complex metrics.

Core Concept of CUPED

Variance Reduction in CUPED: CUPED is fundamentally a variance reduction technique. It aims to reduce the noise in the measurement of the treatment effect, making it easier to detect a true effect if one exists. This is crucial for experiments where the expected effect size is small or the outcome metric is inherently noisy.
Leveraging Pre-Experiment Data in CUPED: The technique relies on a strong linear relationship between a pre-experiment covariate (X) and the post-experiment outcome (Y). By using this covariate, CUPED adjusts the observed outcome, effectively removing the variance explained by the covariate from the outcome variable. This leads to a more precise estimate of the treatment effect.

Mechanism of CUPED

Covariate Selection: The effectiveness of CUPED heavily depends on selecting a covariate that is highly correlated with the outcome variable and is measured before the experiment begins. Often, a user's historical average of the outcome metric (e.g., average daily spend) serves as an excellent covariate.
Adjusted Outcome Calculation in CUPED: For each experimental unit, the adjusted outcome (Y_adjusted) is calculated as Y_observed - θ * X, where Y_observed is the post-experiment outcome, X is the pre-experiment covariate, and θ is a coefficient estimated to minimize the variance of Y_adjusted. This adjustment removes the predictable variation due to baseline differences.

At a glance

Executive summary

CUPED is a statistical method that makes A/B tests more accurate and efficient. It works by using data collected before an experiment to reduce the 'noise' in the results, allowing companies to detect smaller impacts from changes faster. This means they can make better decisions with less data and time.

TL;DR

CUPED helps make online experiments (A/B tests) more precise by using past data to filter out random variations, so you can spot real changes quicker.

Key points

Uses pre-experiment covariate data to reduce the variance of treatment effect estimates in A/B tests.
Solves the problem of low statistical power and long experiment durations by increasing sensitivity.
Widely used by tech companies (e.g., Microsoft, Google, Meta) for product and algorithm evaluation.
Unlike simply increasing sample size, CUPED statistically controls for baseline differences to reduce noise.
Growing application in complex ML evaluations, such as conversational AI, to improve A/B testing robustness.

Use cases

Evaluating website design changes to detect small lifts in conversion rates or engagement metrics.
Optimizing recommendation algorithms by more precisely measuring their impact on user clicks or time spent.
Assessing new search ranking features to quantify subtle effects on query success rates or user satisfaction.
Improving the sensitivity of A/B tests for conversational AI models to detect nuanced changes in user satisfaction or task success [2601.18253v1].
Testing pricing strategies to accurately measure their impact on revenue per user, controlling for historical spending habits.

Also known as

Controlled-experiment Using Pre-Experiment Data, variance reduction technique