The reproduction contract is published before the first live-sandbox or confirmatory run: benchmark reproduction guide →
OpenAI stopped reporting SWE-bench Verified (February 23, 2026) after an audit found gains “increasingly reflect how much the model was exposed to the benchmark at training time,” and retracted its SWE-bench Pro recommendation (July 8, 2026) after flagging roughly 27–34% of public tasks as broken. A benchmark whose hypotheses, metrics, and analysis are written down after the runs is an instrument for confirming whatever its author hoped.
Evidence: Verified evidenceTechnical provenance
Disclosed S-REF development pilots predate the freeze and cannot support scientific claims. Hypotheses, matrix, metrics, and analysis rules freeze before any live-sandbox observation; adapter and baseline operationalizations, artifact hashes, and environment pins freeze before confirmatory execution. Frozen content is append-only; amendments are numbered and stamped.
The Stage-B design requires the held-out fault-seed split to be sealed as complete, labeled artifacts, with ciphertext and plaintext root hashes committed before confirmatory execution. That human-gated sealing act has not occurred. Once sealed, the split will govern the publish threshold and kill criterion without entering the repository or an agent context.
At least 5 seeds per cell (10 planned); confidence intervals and effect sizes, never point estimates; every executed cell reported; INVALID runs retained and marked, never deleted; second-machine reproduction and independent recomputation before any public claim. BetterBench self-scoring, Datasheets, and Croissant metadata ship with results.
Evidence: Design specificationTechnical provenance
Every rung is pre-specified in the current draft. The preselected primary comparator is the composite B5+B3+B6 — the strongest realistic conventional stack — with B5 reported alongside; superiority must reject against both. The ladder is rendered whole, residual failure modes included.
Scroll table horizontally
| # | Baseline | Solves | Residual failure mode |
|---|---|---|---|
| B0 | No protection | nothing | all duplicates / orphans / lost |
| B1 | Model-argument hashing | identical-retry dedup | breaks under re-synthesis (different args) |
| B2 | Agent-generated idempotency keys | some retries | key changes when the model regenerates it |
| B3 | Stable workflow-issued operation IDs | caller-side identity | destination still may not honor the key (C2) |
| B4 | Destination-native idempotency | C1 duplicates | unavailable on C2/C3 |
| B5 | Durable runtime + native idempotency | C1 + own-state exactly-once | punts external APIs to idempotency keys; no C2 adjudication |
| B6 | Provider-native status check / reconcile | some detection | not general; varies by provider |
| B7 | Model-assisted semantic matching | flags likely dupes | probabilistic; unsafe as sole authority |
| R | Proposed system (Irrevon) | C2 detection + safe re-dispatch + orphan/lost surfacing | none beyond destination capability (C3 unsolvable) |
Evidence: Design specificationTechnical provenance
Verbatim from the draft preregistration (§1, implementing master doc §8.6 as amended by AM-14):
“the kill fires iff, for every primary metric on the confirmatory stratum, either (a) the preselected composite comparator is statistically equivalent to R — TOST against the §5.3 margin, never CI overlap — or (b) it is statistically better than R; and (c) no confirmatory cell shows the comparator worse than R by at least the worst-cell gate (§5.3) — a pooled equivalence coexisting with a large local regression is reported as inconclusive with a localized signal, not as a kill and not as a win. If the kill fires, Irrevon is unnecessary and the project is reframed as a teaching artifact (master doc §14). The benchmark is explicitly not designed so the system must win; clause (b) exists so a baseline that beats R kills at least as surely as one that ties it.”
On destinations with dependable native idempotency, Irrevon is expected to show no advantage on duplicate rate. That null is a registered hypothesis (H0-C1) and will be reported as prominently as any win.
Evidence: Testable hypothesisTechnical provenance
On opaque destinations every method — including Irrevon — must fail to detect lost and orphaned effects. The boundary is documented, not excused.
Evidence: Design specificationTechnical provenance
Evidence: Testable hypothesisTechnical provenance
An arm cannot change the denominator it is judged by, and no metric rewards adopting Irrevon’s internal data model.
Scroll table horizontally
| Metric | Definition |
|---|---|
| Duplicate-effect rate | surplus destination effects per true intent ÷ intents eligible for dispatch (oracle-fixed denominator; never computed from finding counts) |
| Orphaned-effect rate | destination effects with no corresponding true fixture intent ÷ total destination effects at read-back |
| Lost-legitimate-effect rate | legitimate intended effects absent at read-back ÷ legitimate intended effects |
| False-suppression rate | legitimate effects blocked ÷ legitimate effects |
| Unresolved-ambiguous rate | AMBIGUOUS unresolved at run end ÷ dispatched (descriptive, arm-conditional) |
| Precision / recall | vs the oracle, per classification (descriptive) |
| Human-review rate & mean time | escalations ÷ effects; wall-clock per escalation |
| Time-to-detect | dispatch → detection interval |
| Compensation correctness | correct compensations ÷ compensations attempted (N/A when none) |
Evidence: Design specificationTechnical provenance
The full preregistration — hypotheses, strata, analysis plan, invalid-run rules, holdout protocol — is a public document in the repository, labeled DRAFT until a human freezes it.
Every public claim distinguishes verified evidence, evidence-backed analysis, testable hypotheses, design specifications, and open questions. Each claim links to the detailed source record.
Evidence: Verified evidenceTechnical provenance
arXiv preprint → agents/reliability workshop → NeurIPS Datasets & Benchmarks track, only if the preregistered criteria and independent recomputation hold. No preprint exists today.
Evidence: Design specificationTechnical provenance