Say your dashboard credits the launch email with 80 sales (a made-up figure). Some of those people were going to buy anyway, and the email was simply the last thing they clicked. Attribution tells you who touched the sale. Incrementality tells you whether the sale would have happened without that touch.
A holdout test measures that gap. You deliberately withhold a campaign from a random slice of your audience, then compare what the two groups do. Whatever extra the exposed group does is the campaign's real effect.
What Google offers, and why most readers can't use it
Google Ads' Conversion Lift studies split users into a group that can see your ads and a holdback group that can't, then report incremental conversions, incremental cost per acquisition and incremental ROAS. The help page lists Display, Search, Video, Demand Gen, App and Performance Max as supported. As of 24 October 2026, the entry bar is still high: a minimum campaign budget of $5,000 USD, at least 1,000 observed conversions, and each campaign can sit in only one lift study at a time. Google recommends running for more than 14 days. The page also says to contact your Google account representative to use it.
Trade press reported the Search and Performance Max expansion in early October (PPC News Feed, 7 October 2026). That followed a Google note in November 2025 saying experiments that once might have cost upwards of $100,000 could now be done for $5,000.
A solo builder doing, say, 40 sales a month doesn't have 1,000 conversions or $5,000 for one campaign. The method itself doesn't depend on Google's tooling, so you can borrow it.
The idea in one paragraph
Pick a campaign. Randomly split the audience before anything goes out. One group gets the campaign, the other (the holdout) gets nothing different from normal. Fix the length of the test in advance. At the end, compare the conversion rate of each group. The difference, multiplied by the size of the exposed group, is your incremental result. Divide that by what your attribution tool claimed and you get an incrementality factor for that channel. Google's own guide to experimentation frames it the same way: compare people who saw your ad with people who were eligible to see it but didn't.
Four holdouts you can run with no special tools
1. Email promo holdout (easiest, most precise). Export your list, randomise it, and leave 10–20% out of a promotional send. You control exactly who receives what, so this is the cleanest test a small business can run.
2. Switchback (on/off weeks). For a channel you can't target by person, such as a podcast sponsorship slot or a small always-on ad budget, alternate weeks on and off. Run at least four weeks of each, decided by coin toss in advance rather than by whichever week feels slow. Compare total sales in on weeks against off weeks.
3. Geo holdout. For paid social, turn off delivery in a few matched regions (countries, states, or cities with similar past sales) and leave it running elsewhere. Compare the change in sales from your pre-test baseline in each set. It suits builders already selling in several markets.
4. Sponsor or partner holdout. Ask a weekly sponsor or affiliate to skip one or two randomly chosen issues: a switchback on someone else's channel.
Step by step: the email holdout
Pick one campaign and one outcome. For example, "purchases of the course within 7 days of the launch email". Write it down before you start.
Export the list to a spreadsheet, one subscriber per row.
Randomise. In column B, enter
=RAND(). In column C,=IF(B2<0.2,"holdout","send"). Then copy column B and paste it as values, becauseRAND()recalculates every time the sheet changes and your groups will shuffle.Check the split. The holdout should be close to 20%, with similar past purchase rates in both groups.
Send to the "send" group only. Tag or segment the holdout in your email platform so no other campaign reaches them during the test window.
Wait the full window. Don't stop early because the numbers look good.
Count purchases in each group by matching buyer emails against the two lists, whether or not the buyer clicked the email.
Calculate lift and the factor, then log it in your reconciliation sheet.
How big does the holdout need to be?
Evan Miller, whose sample size calculator is a standard reference for A/B tests, gives a rule of thumb in How Not To Run an A/B Test: you need roughly n = 16 × σ² ÷ δ², where δ is the smallest difference you care about. When comparing two groups, read n as the size of each group. For a conversion rate p, σ² is p × (1 − p).
Illustrative example, made-up numbers. Your normal purchase rate is 2% and you want to detect a 1 percentage-point lift. σ² = 0.02 × 0.98 = 0.0196. n = 16 × 0.0196 ÷ 0.0001 ≈ 3,136 per group. A 20% holdout of 3,136 implies a list of about 15,700.
If your list is smaller, you have three honest options: hold out a bigger share (30–50%) on lower-stakes sends, test only campaigns you expect to have a large effect, or repeat the same holdout across several sends and pool the results. What you shouldn't do is run a tiny test and treat the answer as settled.
The template
Randomisation sheet: Email | RAND (pasted as values) | Group | Purchased in window (Y/N).
Results block:
Send rate = buyers_send / n_send
Holdout rate = buyers_holdout / n_holdout
Lift (points) = send rate - holdout rate
Incremental = lift x n_send
Factor = incremental / attributed conversions
Illustrative example, made-up numbers. A list of 20,000; 16,000 sent, 4,000 held out. 320 buyers in the send group (2.0%), 48 in the holdout (1.2%). Lift is 0.8 points, so incremental buyers ≈ 0.008 × 16,000 = 128. Your email tool attributed 200 sales to the send. Factor = 128 ÷ 200 = 0.64. Roughly two in three attributed sales were caused by the email.
Decision rule, written before you start: for example, "If the factor is under 0.3 across two tests, cut this campaign's frequency in half; if over 0.7, add a second send." If you'd like a quick significance check in the same sheet, a two-proportion z-score of more than about 1.96 corresponds to roughly 95% confidence.
Reconciliation sheet: one row per channel with Channel | Attributed conversions | Measured incremental | Factor | Test dates | Notes. Multiply each channel's attributed numbers by its factor for a truer picture of where sales come from.
Metrics to watch
Conversion rate in each group, not just clicks or opens
Revenue per subscriber in each group, since discounts can lift orders but cut margin
Unsubscribes and complaints in the send group
For switchbacks and geo tests, the change against each period's or region's own baseline
Pitfalls
Samples too small. The commonest failure. Do the sample-size maths first.
Contamination. Holdout subscribers who see the promo on social or get a forwarded email blur the result. Note it, and keep the holdout out of other sends.
Seasonality. Switchback weeks that line up with paydays, holidays or a viral post will mislead you. Randomise which weeks are on.
Running too short. Give it at least the length of your usual buying cycle. Google itself recommends more than 14 days for its studies.
Peeking. Checking daily and stopping when it looks significant inflates false positives, as Miller's article shows. Fix the end date.
Variations
Always-on holdout: keep 5% of new subscribers out of your welcome sequence permanently to measure what onboarding is worth. It pairs well with a 30-day upsell window.
Cancel-flow holdout: show your retention offer to 80% of cancelling users and measure saves against the 20% who didn't see it.
Start with your next promotional email. The cost is the extra sales that a fifth of your list would have made from one send, and in return you learn what that send is actually worth.