How PRAPI runs pre-registered research on pitch outcomes

The arc

Every PRAPI research cycle ships in two parts. Part 1 is a descriptive report: survey-based findings from practitioners. Part 2 is an experimental report: a controlled A/B test of claims surfaced in Part 1.

The experimental arc turns PRAPI into a system of record for what actually works in B2B PR. Voice validator stops being “we think this matters” and becomes “we ran N controlled tests, formula-enforcement produced X% higher coverage rate.”

Cycle 1 hypotheses

Three hypotheses are pre-registered for Cycle 1 (full pre-registration: /research/preregistration/cycle-1):

• H1, Deadline + concrete data point. Pitches that name a specific external deadline AND a brand-internal numeric claim outperform pitches stripped of both.
• H3, Founder voice, first person. First-person founder framing outperforms third-person agency framing.
• H5, Operational insider data. Claims sourced from the responder's own systems outperform publicly available industry statistics.

How A/B drafting works

1. A consented user generates a pitch through PRAPI. Behind the scenes the voice validator produces two drafts: Variant A (treatment) applies the hypothesis; Variant B (control) strips it. Both drafts persist with a shared variant pair id + a heuristic treatment-isolation check.
2. The user picks one to send (or sends both to different journalists on the same beat). PRAPI logs which version shipped and to whom.
3. Outcomes (reply, quote, published, ignored) accumulate against the pair. The reply-attribution webhook stamps replies automatically; quote and published require operator confirmation.
4. At cycle close + 14 days, outcomes freeze. The analysis cron computes per-arm composite engagement scores (1·reply + 3·quoted + 5·published), Welch's t-test, bootstrap CI, Cohen's d, and an advisory verdict.

Consent and your control

• Opt in is explicit. Manage at /pr-pitch/settings#research. Default for new users is opted-out.
• Opt out preserves history. Pre-registration commits to keeping historical opted-in rows in the corpus; deleting them post-hoc would corrupt sample sizes. After opt-out, no future pitches are captured.
• Per-pitch exclusion. Flag any specific pitch (and its A/B partner) as not-for-research from the outcome capture queue. Excluded rows are kept for compound-effect post-hoc analysis but are removed from the headline aggregate.
• Aggregate-only publication. Per-arm summary statistics ship with each report; row-level pitch data never leaves PRAPI.
• Negative results publish too. If a hypothesis is refuted at n ≥ 200/arm with effect below the 30% MDE, it publishes as a negative result. Hiding failed hypotheses kills the long-term moat.

Anonymization

Reports publish per-(hypothesis × variant × cycle) summary statistics: sample size, composite mean, 95% CI, p-value, Cohen's d, empirical power. The published dataset is aggregate-only. Individual users, brands, and journalists are never identified without explicit per-pitch consent.

Confounder disclosure ships with every Part 2 report: per-arm distribution across 10 dimensions (user expertise, brief quality, journalist relationship, industry vertical, geography, send time, brand age, prior PRAPI volume, outlet tier, time-since-last-pitch). When any dimension shows a materially imbalanced split, the report stratifies the analysis and reports the sub-effect.

Sample size and significance

Per-arm sample-size floors per the pre-registration:

• n ≥ 100/arm: preliminary findings (direction reported, no significance claim).
• n &geq; 200/arm: significance tier; a relative lift &geq; 30% with Bonferroni-corrected p < 0.05 validates the hypothesis.
• Plus: &geq; 30 unique users, &geq; 10 unique outlets, &geq; 30 days of activity per cycle.

Validated hypotheses re-test every 4 cycles (~annually) to catch landscape drift. Refuted hypotheses retire to the archive.

What we do not do

• Optional stopping. Cycles run to their pre-registered close + 14-day freeze; no early stopping when results look favorable.
• Cherry-picking. Pre-registration locks hypotheses, MDE, sample-size thresholds, and confounder list before any data is collected. Amendments are logged publicly with timestamps.
• Burying nulls. Refuted hypotheses publish with the same disclosure depth as validated ones.
• Row-level publication. Aggregates only; individual pitches stay private.

References

• Pre-registration: /research/preregistration/cycle-1
• Part 1 descriptive methodology: /methodology
• Repo (open-source analysis pipeline): github.com/Startvest-LLC/marketing-agent