The two examples below come with saved inputs, outputs and checks that use only the Python standard library and make no network or model call. The evidence files are served with this page under evidence/. Download that receipts directory with its layout intact, then run the commands from its root; no neighboring index.html is required.
The original refund files and CBC saved-margin recount match research commit 888862c9, on branch review/evaluation_manifesto_v1. That research repository requires access; the public evidence files linked here are sufficient for the checks below. The compressed source panel and portable panel checker were added in this revision and retain their own author-repository pin and content hashes.
1. The finite refund model
What it is. An authored, exact finite model: four worlds crossing committed/uncommitted refund state with idempotent/non-idempotent provider behavior, a fifth world with a late-settling original request, three policies (stop, retry, look-then-act), and a census of every delivery sequence of length one to four over two intent identifiers and two authorization states, run against five worker variants and three separately expressed obligations.
What the essay takes from it, and where to find each number in result.json.
| Essay statement | Location in the saved output |
|---|---|
| Cached ledger read leaves four worlds | explorer_scenarios.cached.compatible (4 entries) |
| Fresh read leaves two worlds; at zero, retry and look-then-act pass; at one, stop and look-then-act pass | explorer_scenarios.fresh0.policy_results, explorer_scenarios.fresh1.policy_results |
| Look-then-act is the only policy eligible across all four original worlds | explorer_scenarios.timeout.policy_results (inspect_then_act: 4 passing, 0 failing) |
| Fifth world: fresh reading zero, retry, original settles, two effects | outside_model_witness.executions.inspect_then_act.trace (committed 0 → 0 → 1 → 2) |
| 340 sequences × 5 workers = 1,700 rows; safety-only checker falsely accepts 440 | property_census.case_count = 340; property_census.safety_only_false_accepts = 440; by variant: dedupe_amount 130, refuse_all 310 |
The canonical claim in the essay and its assumptions. For an authorized intent, an exact authoritative reading shows zero effects; if no earlier, concurrent or other new request for that intent can commit after the read, and one retry completes once before the stated horizon, exactly one refund occurs within that horizon. Exact observations, authorized intent, one effect per successful retry and the outcome horizon are rules of the authored transition semantics. The fifth world violates the no-later-commit assumption on purpose.
What it does not establish. Equivalence to any real payment provider; automatic detection that the world family is inadequate (the original family still declares retry eligible after a fresh zero, outside_model_witness.retry_result_within_incomplete_family; adding the fifth world yields mixed outcomes; the separate explorer_scenarios.empty adequacy-failure scenario is supplied directly); a field error rate (the 440 rows are an authored census, not incidents); independent specification (obligations and workers share authorship).
Files. run.py · result.json · REPORT.md
The historical REPORT.md is preserved verbatim. Its secondary links to baseline records, scientific foundations and earlier verification scripts require the original research checkout at the commit above; those ancillary files are not included in this small bundle. The script and output needed for the following rerun are included.
Rerun.
python3 evidence/finite-assay/run.py --output /tmp/finite.json
diff /tmp/finite.json evidence/finite-assay/result.json
The saved output reproduces byte-identically. An independent recount of the 130/310 split from input predicates is in the repository at nap/meeting-prep-2026-09-05/qa/check_finite_explorer.mjs and in article2-algebraic-check.json.
2. The judge-calibration tie recount
What it is. A recount of the saved 700-row human-anchor sample published with CBC v2 at author repository revision 37d50056. The verified computation is this recount of the published margins, not a rerun of the authors' model-score collection.
Pinned author files. Analysis script (double-centering at lines 130–134; strict > 0.0 threshold at lines 442–443) · Saved human-anchor sample · Reported summary. Our copies matched those files byte-for-byte when downloaded at that revision.
What the essay takes from it.
| Essay statement | Exact result |
|---|---|
| Reported gain 68.7% → 76.6% on 700 items | raw strict-positive 481/700 = 68.714%; corrected strict-positive 536/700 = 76.571% |
| 55 verdicts changed; all had raw margin exactly zero | 55 flips; all_changed_verdicts_had_raw_margin_zero: true |
| Corrected margins on those 55 are below 10⁻¹⁶ | between 1.11×10⁻¹⁷ and 8.88×10⁻¹⁷ |
| Treating them as ties gives 481/700 for both | identical at every tolerance tested: 10⁻¹⁶, 10⁻¹⁵, 10⁻¹⁴, 10⁻¹³, 10⁻¹², 10⁻¹¹, 10⁻¹⁰ |
Why the residue is arithmetic, not signal. The correction is double-centered across backbones, so for each item its mean over a complete evaluator panel is exactly zero, and subtracting it cannot change that item's panel-mean margin in exact arithmetic. In finite precision, cancellation can leave a tiny nonzero residue, which the strict threshold then counts as a win.
Claim boundary. This concerns the reported human-anchor improvement in the published artifact. It does not dispose of the paper's rank-consistency results. The source-panel check below confirms that all 55 changed verdicts came from complete, nonmissing five-evaluator panels with integer margins summing to exactly zero. The three rows with larger margin differences each have one missing evaluator margin and do not change a verdict. The cause of the missing provider scores and the original model-score collection remain unverified.
Files. cbc-anchor-check.py · cbc-anchor-sample.csv · cbc-anchor-check.json · cbc-root-crosscheck.json · Audit note
Rerun.
python3 evidence/review/cbc-anchor-check.py
The recount script reads the CSV beside it, checks the saved-margin counts and tested tolerances, and rewrites cbc-anchor-check.json.
Check the underlying evaluator panels
The author's source panel contains 52,500 evaluator rows. We package its original 6,561,072 bytes as a 329,980-byte gzip file. The portable checker decompresses it in memory and verifies the original Git blob 047fc2ce03acd4115675caf76d9ed529811de3af and SHA-256 before examining any rows. It also verifies the SHA-256 of the same 700-row anchor sample used above.
| Check | Verified result |
|---|---|
| All 55 changed verdicts | Five distinct evaluators, no missing scores or margins; each source margin equals chosen score minus rejected score |
| Arithmetic before correction | All source margins are integers and each of the 55 panel sums is exactly zero |
| Three larger saved margin differences | Five evaluator rows each, but only four nonmissing margins; none changes a verdict |
Files. Panel checker · Compressed source panel · Saved panel-check result. The checker reuses the anchor sample above.
python3 evidence/review/cbc-panel-bundle-check.py --output /tmp/cbc-panel-check.json
diff /tmp/cbc-panel-check.json evidence/review/cbc-panel-bundle-check.json
Alternative raw-CSV route. The bundled compressed panel above is the default and is served by the hosted receipts page. To check the author's pinned raw CSV instead, save these four files in one directory: cbc-panel-bundle-check.py, cbc-panel-bundle-check.json, cbc-anchor-sample.csv, and the pinned raw source CSV saved as pilot_complete_panel.csv. Then run:
python3 cbc-panel-bundle-check.py --panel pilot_complete_panel.csv --sample cbc-anchor-sample.csv --output cbc-panel-rerun.json
diff cbc-panel-rerun.json cbc-panel-bundle-check.json
The checker verifies the CSV's Git blob and SHA-256 before analysis, so a wrong or altered download fails loudly. This flat layout was tested and reproduces the saved result byte-for-byte.
This is a locally authored verification of pinned saved data, not author code or a rerun of provider requests. The checker SHA-256 is 26e3034ab26fa3430842f99caa7f03e2cdc809ed96c623d9d983c37774da05b7. The result records the original panel and anchor hashes, all 55 item-level checks, and the three missing-margin rows. Hashes establish which artifacts were checked; they do not establish why a provider score was missing.
3. External sources cited in the essay
| Source | Used for | Boundary |
|---|---|---|
| Dynamic safety cases for frontier AI | Prior art: claims and evidence revised as systems change | Credited, not reproduced |
| Active Evaluation Acquisition | Prior art: choosing informative evaluation items | Credited, not reproduced |
| Science sandboxes | Score optimization can diverge from hidden-rule recovery | Transfer of recovered rules is our hypothesis, not its result |
| Stopping and routing judge panels | A formalized stopping decision for a fixed instrument set | Extension to generated experiments is ours |
| Clean Engineering, Unstable Measurement | Repeatability failures in tested black-box observers | Author-reported preregistration; concerns its setups |
| CBC v2 | Reproducible pipeline with a tie-handling flaw | See §2 |
| Red Queen Gödel Machine v2 | Epoch-frozen criterion; dependent records lose standing | Epoch-local guarantees |
| Dwork et al. 2015 | Uncontrolled adaptive holdout reuse can invalidate guarantees; protocols exist to control it | No implementation claimed |
| S3Gym | Self-judgment and later improvement were not reliably associated | Association, not intervention |
| Naur, Programming as Theory Building | The theory lives in people; no document carries it | Faster reconstruction from the record is our hypothesis, not his |