Causal Impact for Product Data Science

This year, teaching my 2026-2027 data science cohort, I asked students to pick a topic and present it to the class. One topic sat untouched: causal impact.

I don’t really blame them. Next to it on the sign-up sheet were RAG and other LLM-related topics — flashier, more resume-friendly, easier to demo. “Why marketing spend didn’t cause the DAU bump you think it caused” is a much harder sell to a room full of people who came here to build cool things.

Nobody ended up picking it. But causal impact is quietly one of the most important skills in product data science, so I decided to write it up myself instead. Here’s roughly what I’d have wanted them to hear.

Correlation Isn’t Causation (Everyone Knows It, Until It’s Their Dashboard)

It’s a cliché precisely because it’s so easy to forget when the chart is your own. Two lines climb together, and the story writes itself.

The chart alone can’t tell the two apart. That’s the entire problem this post is about.

The First Time It Bit Me: A Campaign With No Attribution

I first ran into this for real at Zynga. Marketing ran a campaign — broad, not targeted at a specific randomized slice of users — and there was no attribution tracking set up ahead of time. No control group, no holdout, no click-through tagging. Just a campaign that went out, and a DAU line that went up shortly after.

Naturally, marketing wanted credit for the lift. And naturally, my job was to figure out how much of it, if any, was actually theirs.

When A/B Testing Isn’t On the Menu

The gold standard for causal impact in industry is the randomized controlled experiment — the A/B test. Randomize who’s exposed, compare the groups, done. It’s clean because randomization handles all the confounders for you, known and unknown.

The problem is that a lot of real interventions can’t be randomized at the user level: a national marketing campaign, a press cycle, a pricing change announced publicly, a platform-wide feature launch. There’s no “control group” sitting inside the same market at the same time, untouched.

When that’s the situation, people reach for a small set of quasi-experimental methods instead.

Difference-in-Differences, Briefly

The most common of these is difference-in-differences (DiD). The idea: find a market or segment that wasn’t exposed to the treatment but otherwise moves similarly to the one that was. Compare the change in the treated group before vs. after, and net out the change in the control group over the same window. What’s left is attributed to the treatment.

The whole method leans on one assumption: parallel trends — that absent the campaign, the treated group would have kept moving in step with the control group.

DiD is simple, explainable to a stakeholder in one slide, and a very reasonable first move. But it needs a genuinely comparable control group, and one control group is a small sample size of exactly one.

Enter Bayesian Structural Time Series

This is the part I most wanted covered, because it’s what I reached for at Zynga and what shows up again and again in product DS: Bayesian structural time series (BSTS).

The general idea it implements has a name: the synthetic control. Instead of finding one perfectly matched comparison group and taking a single before/after difference like DiD does, you construct a synthetic version of the treated market — built from one or more untreated comparison series — that approximates what the treated market would have looked like without the intervention. BSTS is one way to build that synthetic control: it combines a local trend and seasonal component with a regression on other time series that are correlated with your target but weren’t touched by the treatment. Feed it enough good comparison series, and it produces a synthetic counterfactual — a full predicted curve, with uncertainty, for “what DAU would have been.”

The effect estimate is then just observed minus predicted, at every point in time, with a credible interval instead of a p-value.

Google’s CausalImpact Package

The industry-standard tool for this is Google’s CausalImpact R package, built on top of bsts. It’s become the default way product and marketing teams estimate the effect of something they couldn’t A/B test — a launch, a PR moment, a campaign.

This is the package I used to try to untangle the Zynga campaign. The main market had run the campaign, so I needed a comparison series to build a synthetic control from — a market that hadn’t been touched by the campaign, but that moved with the main market for reasons unrelated to marketing. Australia was the pick, based on business acumen more than a formal test: we knew from experience it behaved similarly to the US market. But the one non-negotiable requirement was simpler than any of that: the campaign did not run there. So I fed its DAU into CausalImpact as a covariate and let the model build the synthetic control — the predicted counterfactual — for what the main market’s DAU would have done on its own.

The Honest Take: It’s Not Magic

Here’s the part I made sure got said out loud in class, because it’s the part people skip when they cite this package: the results were not great, and it was often genuinely hard to get anything statistically significant out of it.

A few reasons why, in my experience:

• The counterfactual is only as good as your synthetic control. Australia being untouched by the campaign was certain; Australia behaving like the US market was a judgment call based on business acumen, not a validated fact. A comparison market that’s subtly different in the wrong way quietly biases the whole synthetic control, and everything built on top of it.
• DAU is noisy, and marketing lifts are often small relative to that noise. A few percentage points of lift can be real and still sit inside a credible interval that touches zero.
• The model is doing a lot of work with a short post-period. Right after launch, you simply don’t have much data to distinguish “real effect” from “this week was unusual anyway.”
• Everything else that’s happening at the same time doesn’t stop. Other campaigns, product changes, competitor moves — BSTS assumes your control series absorbs all of that. It doesn’t, fully.

In the Zynga case specifically, the credible interval on the cumulative effect kept widening the further out from launch I looked, and by the end of the post-period it had drifted back down to touch zero — the band in that last chart above isn’t exaggerated for effect, that’s roughly the shape of what I actually got back. Statistically, I could not say with confidence that the campaign moved DAU at all.

But here’s the thing: that wasn’t a failure of the method, it was the honest ceiling of what was answerable. There was no attribution tracking, no holdout, no randomization — a wide credible interval was the correct amount of uncertainty to have, not a sign I did it wrong. Given what we actually had to work with, “here’s our best estimate, and here’s how much you should trust it” was the best possible answer, and a far more defensible one than the confident-sounding, made-up-precision number marketing wanted to put in a slide.

Why This Belongs in the Curriculum Anyway

Most of the flashy topics on that sign-up sheet assume you got to design the experiment. Product data science is full of moments where you don’t: a brand campaign, a legal or policy-driven change, a one-time event, a competitor’s move you have to react to after the fact. In those moments, knowing how to construct a defensible counterfactual — and being honest about how much uncertainty is still in it — matters more than knowing the newest model architecture.

Maybe next time it’s on the sign-up sheet, someone will actually pick it.