Hey,

A founder told me something a couple of weeks ago that I keep coming back to.

They sell online and through their own stores. At one point they turned the ads off.

Online sales fell. Everybody expects that.

But then... the shops went quiet too.

 

Sit with that for a second. It's the most valuable number in their business, and it exists by accident.

No report inside Meta or Google can produce it. The platforms simply measure something else.

A platform measures attribution. It tells you which of the people who bought had touched an ad first.

Incrementality asks a different question. What would have happened if the ad had never run?

To answer that you need a version of the world without your ads in it. The platform has no access to that world.

You do.

 

The experiment you've already run

Most brands have run this test by accident at least once.

A card expired and spend stopped for four days. An account got flagged over a weekend. You paused everything during a stock-out. Somebody fat-fingered a budget.

Every one of those is a holdout period sitting in your Shopify data right now. Almost nobody goes back to read it.

Which is a shame, because it already cost you the revenue. You may as well collect the information you paid for.

Go and find yours.

Pull the dates where paid spend dropped to near zero for three days or more in the last eighteen months.

Then look at total revenue and new customer orders across those days. Total, not platform-attributed.

If revenue fell roughly in line with the spend, your ads were doing real work.

If it barely moved, a meaningful share of what your platforms claimed was demand that existed anyway.

That second outcome is more common than anybody wants. It's most common in brands with strong organic and strong repeat.

One caveat before you read too much into it. A few dark days tells you a direction, never a number.

Your existing demand carries you through the first stretch, so a short gap always understates what paid was doing. The longer the gap, the more you can trust it, and a considered purchase needs longer again. If people take a month to decide on your product, a four-day pause barely registers.

Treat the accidental ones as a reason to run a proper test, rather than as the test.

 

Why the fashion brand's version was so clean

Their pause moved sales in a channel the ads could not possibly track.

Nobody walking into a shop clicks anything. There's no pixel.

So when store revenue moves with ad spend, you're looking at demand creation no attribution model would ever have credited.

Which cuts both ways. This part is worth holding on to.

Brands with retail, wholesale or phone orders are almost always understating what paid does for them. The platform only sees the slice that ended on a tracked checkout.

So if you have offline revenue and you judge paid on tracked ROAS alone, you've been grading it on a fraction of its output.

 

Running one on purpose

Accidental holdouts are free. Deliberate ones are better.

The geographic version is the one I'd run.

Pick two sets of regions that look alike. Similar revenue, similar order values, similar seasonality. States, countries or metros all work, as long as delivery splits cleanly.

Turn paid off in one set. Leave the other running untouched.

Then hold your nerve for at least four to eight weeks.

Two weeks tells you almost nothing. Your existing demand carries you through the first stretch, and the dip arrives later than people expect.

Measure total revenue and new customers per region, indexed to the four weeks before you started. The gap between the two sets is your incremental contribution.

It's the only version of that number you'll ever own outright.

Everything else you've been shown is a model with your platform's assumptions baked into it.

Three rules decide whether the test is worth anything.

Never run it in your peak. Being wrong in November costs far more than the information is worth.

Never judge it on platform numbers. The whole point is measuring something the platform can't see.

And never run it on a single region. One region is weather, a local competitor, a holiday you forgot about.

 

If you can't split by geography

Plenty of brands can't. You're in one market, or your volume is too concentrated to divide.

The fallback is a time-based holdout. It's weaker, so treat the result as a direction rather than a measurement.

Pick two comparable periods. No promotion, no launch, no seasonal step in either. Run normally through the first, go dark through the second, and compare total revenue indexed to the eight weeks before both.

It's weaker because time carries confounds a geo split removes. Weather. A competitor's sale. A press mention. Your own email calendar.

None of those politely hold still while you test.

One confound is worth checking either way. Watch your branded search volume through the dark period. If branded search falls alongside revenue, your ads were building demand that landed through other channels. And your tracked ROAS never counted it.

 

What the test costs

A holdout costs you real revenue in the region you go dark in. That's the price of the information. Price it before you start.

Rough version: take the region's normal monthly revenue, multiply by the share you'd expect paid to be driving, multiply by the length of the test. That's your worst case.

Then weigh it against the decision it unlocks.

Deciding whether to move your annual budget up or down by a real amount? A four-week holdout in one region out of eight is cheap.

Deciding nothing? Don't run it.

The brands that get the most from this treat it as an annual exercise. Your incremental contribution changes as your brand gets better known, so a number from two years ago is history.

 

What to do with the answer

If incremental contribution comes in far below attributed contribution, your real acquisition cost is higher than you think. Your spend ceiling is lower too.

If it comes in far above, you've been under-spending for a long time. And the case for a bigger budget now has evidence behind it.

Either answer changes what you do on Monday. That's more than most reporting achieves.

And you'll own the number. Nobody gets to revise it in a product update.

 

One timing note. It matters more than the method.

Don't run a deliberate holdout in the next ten weeks. Go and read your accidental ones instead, then schedule the real test for January, when being wrong costs you least.

Say you're past $1M a year and spending $30K a month or more on ads. If you want help picking the regions and reading the result properly, I keep a few free 30-minute calls open each week.

Talk soon,
Andrej

PS: Reply "holdout" and I'll send you the region-matching sheet and the indexing formula. It takes the guesswork out of picking the two groups.

PPS: Still trying to prove it from the dashboard →