Costs
How to rebuild a product positioning benchmark claim and set a stop rule
Rebuild one product positioning benchmark claim from its stated inputs, compare it with your field data and write the stop rule before the claim runs.
What to take away
- Write the decision threshold and the stop rule before you read the vendor's figure.
- Rebuild the headline figure from its stated respondents and yes count. It has to reproduce.
- Set your own field figure beside it and explain any gap before the claim runs.
- Keep the external comparison separate from the internal threshold you will act on.
- A benchmark with no maturity window will make a new product look like a failing one.
- Put the stop rule and the review date in the same file that holds the claim.
Write the rule before you read the figure
Settle the decision rule before you read the vendor's number. Otherwise the figure sets the threshold, and every result looks like a pass. The rule has four parts: population, period, measure and decision.
Four fields before reading numbers
- Populationwho is counted, who is excluded
- Periodstart, end, cohort maturity window
- Measuredefinition, denominator, trade unit
- Decisionthreshold to act on, tolerated uncertainty
- Missing fieldrecord it, do not guess
The worked example is illustrative, not a published benchmark. A vendor claims that 62 percent of mid-market North American software buyers understood a new category claim after one exposure. The four fields for that claim look like this:
- Population
- mid-market North American software buyers, 200 to 2,000 staff, current customers excluded.
- Period
- one quarter, with 90 days of cohort maturity before the claim ran.
- Measure
- share who answered yes to one comprehension question, denominator everyone who saw the claim.
- Decision
- run the claim only if comprehension clears 60 percent and cost per session stays flat.
If any field is missing, the figure is not comparable yet. Say so in the file rather than filling the gap with a guess. A population defined in advance keeps the comparison honest.
Rebuild the vendor's figure from its inputs
Take the one figure the plan leans on and rebuild it from its stated inputs: same population, same denominator, same period, same maturity window. The vendor states 500 respondents and 310 yes. 310 divided by 500 is 0.62, so the claim as published holds.
A figure that reproduces proves the arithmetic and nothing more. It says nothing yet about your buyers. A benchmark that survives a rebuild is evidence, and one that does not is a talking point.
Set it against your own field data
Your own field data shows 240 yes from 500, which is 48 percent. That 14 point gap is the thing to explain, and it also sits below the 60 percent threshold written in advance.
| Source | Yes | Respondents | Share | Clears 60 percent |
|---|---|---|---|---|
| Vendor claim | 310 | 500 | 62 percent | Yes |
| Your field data | 240 | 500 | 48 percent | No |
The rule written first now does its job. The claim does not run on the strength of the vendor's number, and nobody has to argue about it afterwards. The cost-per-session condition stays open until a run produces a figure to test.
Explain the gap before you act
A gap can hide in three places: the population, the maturity window and the definition. Check each one before blaming the claim, because each changes what the figure is a figure of.
If your number lands close to the vendor's, you understand the benchmark. If it lands far off, you have found either an error or a definition you misread, and both are worth knowing before the figure reaches a slide.
Record the largest remaining comparison limit next to the result. A benchmark with no maturity window will make a new product look like a failing one, so ask how long the vendor's cohort had before anyone measured it.
What outside sources can and cannot support
This method does not supply reference values. For published figures, go to a published benchmark set. The Product Marketing Alliance publishes an annual State of Product Marketing report, and Benchmarkit publishes an annual B2B SaaS metrics benchmark.
Both are commercial research, so check the edition, the sample and the definition before you quote a number. Name the source, the edition year and the sample with every figure you use.
The FTC advertising substantiation policy says advertisers need a reasonable basis for advertising claims before they are disseminated. Apply it to the exact promise, audience and qualification you are making.
Rebuilding follows an auditor's habit. The GAO data reliability course lists common steps for an assessment: review existing information, interview staff, test data, use source records and consider information system controls.
Set the stop rule and the review date
Close the file with a short list, so the next reader sees the same frame and the same limit.
- One defined population and one comparison basis
- Message comprehension tracked by segment against cost and quality
- One normal path and one failure path walked
- Claims, inputs, versions and decisions kept together
- What you rejected, and why
- The stop rule and the next review date
Read the result against your own pilot data, and track message comprehension by segment as the pilot runs. A benchmark tells you what a comparable group looked like in a stated window. It cannot tell you what your buyers will do.
