What Is Holdout Test?
A holdout test is an experiment that deliberately withholds advertising from a randomised group (a geography, an audience segment, or users who see a public-service placebo ad) so you can compare their conversion rate to the exposed group and measure real incremental lift. It is the gold standard for proving ads caused conversions, because it removes the assumptions attribution models make.
A geo holdout on a Meta account
A DTC brand pauses Meta ads in 3 mid-sized US metros (holdout) while continuing in 30 matched metros (exposed). Over 4 weeks, exposed metros generate $1.2M revenue; holdout metros, adjusted for population, generate $0.95M. Incremental lift: $250k on $180k spend, incremental ROAS approx 1.4x. Reported platform ROAS during the same period was 4.3x. The gap is the price of not testing.
Benchmarks
- Geo holdout: highest external validity; needs matched-market design and 4 to 8 week runs.
- Audience holdout (Meta Conversion Lift, Google Conversion Lift): easiest to run; requires enough scale to hit statistical power.
- PSA / ghost bid tests: cheapest; comparing exposed vs users who would have been exposed but got a placebo.
- Typical incremental lift below reported ROAS: 20 to 60 percent, higher on retargeting.
Why it matters
Every attribution number is a story about credit; a holdout is the only measurement that answers 'would this conversion have happened anyway?' Brands that run one holdout per quarter (per major channel) usually discover that 20 to 50 percent of reported conversions were not incremental and reallocate budget accordingly. Brands that never test overspend on the wrong channel for years.
Common mistakes
- 1.Running a holdout for one week. Not enough time to overcome noise or purchase-cycle lag.
- 2.Using unmatched control markets. Small metros in different regions have very different baselines; matched-market or synthetic control designs fix this.
- 3.Ignoring cross-channel spillover. If you pause Meta but not Google, part of the 'holdout' revenue is Google picking up the slack.
- 4.Treating one test as the permanent answer. Incrementality shifts with creative, offer, seasonality; re-test quarterly.
Put Holdout Test to work
Free calculators
Related services
FAQs about Holdout Test
How long should a geo holdout run?
4 to 8 weeks minimum. Shorter and you cannot separate the lift from baseline noise and purchase-cycle lag.
How is a holdout different from a conversion-lift study?
A conversion-lift study is a specific type of audience holdout run inside Meta or Google's own tools, using their user pools. A geo holdout is run at the market level, independent of any platform.
How much does an incrementality test cost?
Nothing extra in ad spend beyond what you were already running; the 'cost' is the temporary revenue you may forgo in the holdout markets. On a healthy account this is 1 to 3 percent of quarterly revenue, well worth the clarity.
Can I run holdouts on small accounts?
Yes, but expect noisy results below roughly 300 to 500 conversions per group. Small accounts should focus on MER trend analysis first, then holdouts once volume grows.
Related terms
Measures the lift ads caused vs what would have happened anyway.
Incrementality test that turns spend on/off by region to isolate impact.
How credit for a conversion is assigned across ad touchpoints.
Statistical model measuring channel contribution using historical data.
Enough data to say a result is real, not random noise; usually 95% confidence or higher.
Total revenue ÷ total ad spend, the blended, attribution-free ROAS.
Controlled test measuring incremental awareness or recall from ads.