How PRAPI runs pre-registered research on pitch outcomes

The arc

Every PRAPI research cycle ships in two parts. Part 1 is a descriptive report — survey-based findings from practitioners. Part 2 is an experimental report — a controlled A/B test of claims surfaced in Part 1.

The experimental arc turns PRAPI into a system of record for what actually works in B2B PR. Voice validator stops being “we think this matters” and becomes “we ran N controlled tests, formula-enforcement produced X% higher coverage rate.”

Cycle 1 hypotheses

Three hypotheses are pre-registered for Cycle 1 (full pre-registration: /research/preregistration/cycle-1):

• H1 — Deadline + concrete data point. Pitches that name a specific external deadline AND a brand-internal numeric claim outperform pitches stripped of both.
• H3 — Founder voice, first person. First-person founder framing outperforms third-person agency framing.
• H5 — Operational insider data. Claims sourced from the responder's own systems outperform publicly available industry statistics.

How A/B drafting works

1. A consented user generates a pitch through PRAPI. Behind the scenes the voice validator produces two drafts: Variant A (treatment) applies the hypothesis; Variant B (control) strips it. Both drafts persist with a shared variant pair id + a heuristic treatment-isolation check.
2. The user picks one to send (or sends both to different journalists on the same beat). PRAPI logs which version shipped and to whom.
3. Outcomes — reply, quote, published, ignored — accumulate against the pair. The reply-attribution webhook stamps replies automatically; quote and published require operator confirmation.
4. At cycle close + 14 days, outcomes freeze. The analysis cron computes per-arm composite engagement scores (1·reply + 3·quoted + 5·published), Welch's t-test, bootstrap CI, Cohen's d, and an advisory verdict.

Consent and your control

• Opt in is explicit. Manage at /pr-pitch/settings#research. Default for new users is opted-out.
• Opt out preserves history. Pre-registration commits to keeping historical opted-in rows in the corpus; deleting them post-hoc would corrupt sample sizes. After opt-out, no future pitches are captured.
• Per-pitch exclusion. Flag any specific pitch (and its A/B partner) as not-for-research from the outcome capture queue. Excluded rows are kept for compound-effect post-hoc analysis but are removed from the headline aggregate.
• Aggregate-only publication. Per-arm summary statistics ship with each report; row-level pitch data never leaves PRAPI.
• Negative results publish too. If a hypothesis is refuted at n ≥ 200/arm with effect below the 30% MDE, it publishes as a negative result. Hiding failed hypotheses kills the long-term moat.

Anonymization

Reports publish per-(hypothesis × variant × cycle) summary statistics: sample size, composite mean, 95% CI, p-value, Cohen's d, empirical power. The published dataset is aggregate-only. Individual users, brands, and journalists are never identified without explicit per-pitch consent.

Confounder disclosure ships with every Part 2 report: per-arm distribution across 10 dimensions (user expertise, brief quality, journalist relationship, industry vertical, geography, send time, brand age, prior PRAPI volume, outlet tier, time-since-last-pitch). When any dimension shows a materially imbalanced split, the report stratifies the analysis and reports the sub-effect.

Sample size and significance

Per-arm sample-size floors per the pre-registration:

• n ≥ 100/arm — preliminary findings (direction reported, no significance claim).
• n &geq; 200/arm — significance tier; a relative lift &geq; 30% with Bonferroni-corrected p < 0.05 validates the hypothesis.
• Plus: &geq; 30 unique users, &geq; 10 unique outlets, &geq; 30 days of activity per cycle.

Validated hypotheses re-test every 4 cycles (~annually) to catch landscape drift. Refuted hypotheses retire to the archive.

What we do not do

• Optional stopping. Cycles run to their pre-registered close + 14-day freeze; no early stopping when results look favorable.
• Cherry-picking. Pre-registration locks hypotheses, MDE, sample-size thresholds, and confounder list before any data is collected. Amendments are logged publicly with timestamps.
• Burying nulls. Refuted hypotheses publish with the same disclosure depth as validated ones.
• Row-level publication. Aggregates only; individual pitches stay private.

References

• Pre-registration: /research/preregistration/cycle-1
• Part 1 descriptive methodology: /methodology
• Repo (open-source analysis pipeline): github.com/Startvest-LLC/marketing-agent