Answering the product marketing analytics questions in plain terms

What to take away

Three questions, three different numbers

Most teams ask the same thing: which channel drove the signups? The honest answer depends on which of three claims you are making.

Attribution splits credit among touchpoints using a rule the analyst picks, such as last click or even spread. It is a bookkeeping convention, not a measurement of cause.

The default rule differs by tool. Google Analytics 4 reports on a data-driven model and still offers last click. Meta Ads Manager credits conversions inside a 7-day click and 1-day view window unless you change it.

Correlation shows two series moving together, which is where most dashboards stop. Incremental effect asks what would have happened without the campaign. Only incremental effect supports a budget decision on its own.

Mixing them is the common failure. A dashboard shows a channel with rising spend and rising signups, someone calls it the best performer, and nobody checks whether the signups would have arrived anyway. Write down which claim each number makes before it reaches a slide.

What the evidence has to survive

A result that cannot be re-derived from stored inputs is not evidence. Keep the query, the extract date, the population definition and the version of the metric next to the finding.

The W3C information and relationships guidance is worth reading for a narrower reason than accessibility compliance: it asks whether structure conveyed visually is also conveyed in code. The same test applies to your reporting. If a table's meaning lives only in column position, a reader downstream cannot check it.

Privacy follows the same logic. The W3C Privacy Principles set out shared concepts for system designers and warn against pushing privacy work onto the individual.

Apply that to the actual data flow in your analytics stack, then confirm the governing law with counsel. In Canada that means PIPEDA at the federal level, and Quebec's Bill 96 adds French-language obligations that touch customer-facing reporting.

The GDPR covers the EU, and California's CPRA applies to most consumer businesses above its revenue and record thresholds. Consent tools such as OneTrust and Cookiebot log a choice; they do not decide what you may collect.

Choosing the test design

The NIST experimental design guidance starts with the objective and the practical constraints, not the statistics. That ordering matters. A team that picks a method first usually picks one it cannot run.

For most launch questions the split is simple. Descriptive reporting answers what happened. A controlled comparison answers whether your change caused it. If you cannot hold a control group, say so, and label the finding as descriptive.

Meta Conversion Lift and Google Ads conversion lift studies run holdouts for teams that buy media on those platforms. Both compare exposed and unexposed groups, so both need enough volume to detect the lift you care about.

Sample size, test duration and minimum detectable effect are arithmetic, not opinion. Work them in your own variables: baseline conversion rate, the lift you would act on, and your weekly traffic. Evan Miller's sample size calculator and Optimizely's take those same three inputs and are free to use.

Where you lack a baseline, cite a published benchmark with its date: your analytics vendor's benchmark report, or an industry association's.

A worked example

A team launches a paid pilot in Ontario and sees signups rise 18% during the campaign. Before crediting the campaign, they check three things.

If the answer to the third is yes, they have a comparison basis and can estimate incremental effect. If not, they have a correlation and should say so in the readout.

When no clean holdout exists, Google's open-source CausalImpact package fits a Bayesian structural time series to control regions and estimates what the treated region would have done. It assumes the control regions would have tracked the test region, so check that assumption before you quote the number.

Where the build decisions sit

Tooling choices constrain what you can measure later. The GOV.UK technology selection guidance recommends adaptable choices, data control, security review and attention to total ownership cost. Those questions apply to an analytics stack as much as to a public service.

Data ownership is the one teams skip. If your event data lives only inside a vendor's warehouse, you cannot re-run an analysis after a definition changes. Export paths and retention terms belong in the contract review, alongside the security assessment.

Pricing models shape that choice too. Amplitude, Mixpanel, Heap and PostHog bill on tracked users or events, so a launch that spikes traffic also spikes the bill. PostHog and GrowthBook can be self-hosted, which keeps raw events on your own servers.

A warehouse such as Snowflake or BigQuery holds the raw events. A transformation tool such as dbt rebuilds the metric when the definition changes, and a BI layer such as Looker or Tableau reads from it. That chain is what makes a number reproducible a year later.

For the measurement plan that sits on top of these choices, see how to measure product marketing analytics. The wider discipline is covered under product marketing analytics.

Common questions

What is the difference between attribution and incremental effect?

Attribution assigns credit by a chosen rule. Incremental effect estimates what would have happened without the campaign, usually through a control group or a holdout. Only the second supports a claim that the campaign caused the change.

How long should a launch test run?

Long enough to cover a full buying cycle for your product and to reach the sample size your minimum detectable effect requires. Work that arithmetic in your own baseline rate and traffic figures rather than borrowing a rule of thumb.

What should we do when two sources disagree?

Compare definitions first: population, period, event definition and filters. Most disagreements resolve there. If they do not, publish both with their source dates and say which one the decision rests on.

When does a privacy question need a lawyer?

Whenever the answer depends on which law applies, what consent is valid, or how long data may be kept. Name the regulator, document the data flow, and get that call from qualified counsel in the relevant jurisdiction.

Which analytics tool should we start with?

Match the tool to the question. Google Analytics 4 is free and covers web and app traffic. Amplitude and Mixpanel suit event-level product questions but bill on volume. PostHog and GrowthBook can run on your own hardware.

What is a holdout, and when can we use one?

A holdout is a group deliberately left out of the campaign so you can compare it with the exposed group. You need enough of them to detect the lift you would act on. Small audiences rarely support one.

How do we choose a minimum detectable effect?

Start with the smallest lift that would change your decision, not the smallest lift you hope to see. A smaller effect needs a longer test or more traffic. Publish the number before the test starts so nobody moves it after.