Significant Is Not Profitable: Reading an Incrementality Test

Incrementality: The Metric That Proves Marketing’s Impact

Author:

Ara Ohanian

Published:

October 20, 2025

Updated:

November 20, 2026

In July 2025, Haus, which sells incrementality testing, reported in a review of 640 Meta experiments that Meta’s Incremental Attribution setting had a 43% success rate against standard attribution, on a limited sample. In July 2026 it reported a reversal: from July 2025 to June 2026, the setting averaged 1.26 times standard attribution’s incremental ROAS (80% interval 1.11 to 1.43). Its re-analysis of the prior year gave 0.80, range 0.58 to 1.10. That range crosses 1.0, so those tests could not separate the two settings.

Google made Meridian GeoX, its geo testing tool, generally available worldwide in September 2026, and its Bayesian Conversion Lift reports give a probability that a campaign raised conversions and an 80% credible interval, not a yes or no. Our guide to running holdout, geo and platform lift tests covers setup; this post covers the readout.

Significant and profitable are different questions

Each result reduces to incremental ROAS, revenue the ads caused divided by their cost; Google’s geo study example is 2.2, range 1.3 to 3.5. Significance asks whether the range clears zero. Your budget asks whether it clears break-even: 1 divided by your margin after variable costs. The American Statistical Association’s 2016 statement says significance “does not measure the size of an effect or the importance of a result.” A test can be significant and unprofitable, or not significant and still decisive.

A hypothetical example, read in order

In this hypothetical example, a retailer spends $50,000 over four weeks on Meta prospecting in its test regions. The model predicts $1,000,000 of revenue there without ads; actual revenue is $90,000 higher, a 9% lift, so incremental ROAS is 1.8. The 90% interval on incremental revenue, $35,000 to $145,000, puts it between 0.7 and 2.9. With a 50% margin, break-even is 2.0. Meta’s reporting credited the campaigns with $175,000, a platform ROAS of 3.5.

The interval excludes zero, so the ads very likely worked. It straddles 2.0, so the test cannot settle whether they paid. And 1.8 divided by 3.5 gives an incrementality factor of about 0.51, so the platform ROAS that means break-even is 2.0 divided by 0.51, about 3.9. The comfortable-looking 3.5 sits below the line.

When the test and the dashboard disagree

The gap runs both ways. Meta’s own white paper, drawing on 307 studies by 54 advertisers, found rules-based attribution undervalued Meta by 31% at the median: Meta’s research about Meta. In eBay’s field experiment, brand-keyword ads had no measurable short-term benefit. Ask how any platform built its control group, as DoorDash’s ghost-ad holdouts illustrate.

So translate the dashboard rather than argue with it: apply your latest factor, bid to the calibrated target, feed the result into your mix model as a calibration point.

An inconclusive test is not a zero

A pre-launch power analysis gives the minimum detectable effect, the smallest lift the design can reliably catch. Meta’s open-source GeoLift calls it the minimum lift for a well-powered test, at a default 90% confidence level. If that is 12% and break-even needs 10%, a null result does not show the ads lose money; statisticians Douglas Altman and Martin Bland called that “absence of evidence”, not evidence of absence. Also check a vendor’s figure measures lift at all: Epsilon’s “up to 50%” for Snapchat was a match rate.

A budget rule to write down before the test

A sensible rule for allocating paid media budget after a test:

  • Interval wholly above break-even: scale in 20 to 30% steps and retest, since the next dollar usually earns less than the average.
  • Interval straddles break-even: hold spend, bid to the calibrated target, rerun longer or with a larger holdout.
  • Above zero, below break-even: the channel works but is overfunded. Cut toward break-even, or raise the return on the same traffic with conversion rate work on the pages the ads point to.
  • Interval includes zero: check its upper end. Below break-even, cut. Above it, the test was too small to judge.

Significance says the ads did something. The gap between the interval and break-even says what to do with the money.

Frequently Asked Questions

Why is my lift test ROAS lower than Google Ads or Meta reports?

Standard platform attribution credits any conversion that followed an ad within the attribution window, including people who would have bought anyway. A lift test counts only the difference between people or regions that saw ads and those that did not, so the gap is over-credit. Divide incremental ROAS by platform ROAS for the same campaigns and period to get a factor for daily reporting.

What does the interval around a lift result mean?

It is the range of results consistent with the data. In a Bayesian report such as Google Conversion Lift, an 80% credible interval means an 80% probability that the true lift falls inside it. Width matters as much as the midpoint: 0.7 to 2.9 around a break-even of 2.0 settles nothing about profit. More spend, a longer test or a bigger holdout narrows it.

What should I do if my incrementality test is inconclusive?

Check what the test could have detected. If its minimum detectable effect was larger than the lift you need to break even, a null result does not show the ads lose money. If the upper end of the interval sits below break-even, the test has answered you anyway. Otherwise check the conversion data feeding it (Google’s updated Conversion Lift setup guide, reported on November 13, 2026, spells out more detailed data requirements) and rerun with a larger holdout or longer window.

How long does an incrementality result stay valid?

Until something it measured changes: creative, bidding, targeting, attribution settings, pricing or season. The Haus reversal above shows how much one year can move a comparison. A sensible rhythm is to retest the biggest channels once or twice a year and after any major change, and to record the test date beside every factor used in reporting or bid targets.