Data
Everything on this site is rendered at build time from versioned JSON artifacts in a data/ directory, and the same files are served statically under /data/…, so the links below are direct downloads. Every file embeds the run id, bank version, and the bank's SHA-256 content hash; scores are comparable only within a bank hash. There is no API; the JSON files are the interface. The cross-run index is index.json.
Published runs
| Run | Date | Bank | Models | Files | Downloads |
|---|---|---|---|---|---|
| full-2026-08latest | 2026-08-07 | v1.0.0 · sha256:8c9f5828… | 14 | 1,865 | leaderboardmanifestreceipts-manifestmodels/ and receipts/ under /data/runs/full-2026-08/ |
| pilot-2026-08 | 2026-08-06 | v0.1.0 · sha256:a2fb28c8… | 3 | 402 | leaderboardmanifestreceipts-manifestmodels/ and receipts/ under /data/runs/pilot-2026-08/ |
| sample-2026-08synthetic | 2026-08-05 | v0.1.0 · sha256:234a5c51… | 13 | 744 | leaderboardmanifestreceipts-manifestmodels/ and receipts/ under /data/runs/sample-2026-08/ |
- full-2026-08: real evaluation run. 75,696 model calls, 156,068 judge calls, $421.09 total; 1,568 demographic receipt shards.
- pilot-2026-08: real evaluation run. 11,952 model calls, 26,524 judge calls, $13.31 total; 336 demographic receipt shards.
- sample-2026-08: synthetic sample data generated to exercise the full published schema; it does not report on real model behavior. 37,440 model calls, 76,752 judge calls, $146.75 total; 728 demographic receipt shards.
Raw response archives
The working per-run JSONL files (raw model responses and judged rows, one JSON object per line, including error envelopes for permanently-failed calls) are retained by the operator and are not separately published. The published artifacts above carry the same substance in verifiable form: every response and its judge labels appear in the receipts shards, and every file embeds the bank version and hash they were produced under.
Files per run
| File | Contents |
|---|---|
| manifest.json | models, effective params, judge pairs, bank pin, harness git SHA, totals, judge agreement (κ per rubric and per subject vendor, contested rate), run-average control band |
| leaderboard.json | per-model overall + per-attribute consistency with 95% CIs and evidence tiers, control bands, refusal asymmetry, repeat instability, contested rate, n, cost |
| models/<file_slug>.json | per-attribute metrics with CIs, per-control-set disparity, directional lean, pairwise-gap heatmap, per-concept disparities, top-10 most inconsistent receipts |
| receipts/<file_slug>/<attribute>/<concept_id>.json | matched prompts with swap spans, every response repeat, judge votes, final labels, per-group summary (control-set shards included) |
| receipts-manifest.json | shard paths, receipt counts, flip counts, group-pair facet counts |
Schema
The authoritative shape of every artifact is defined in docs/contracts.md (the "Published artifacts" section) and mirrored as TypeScript types in site/types/schema.ts, which this site consumes at build time. The essentials:
- Every per-run file carries the envelope {schema_version, run_id, bank_version, bank_hash, generated_at}; index.json spans runs and carries no envelope. Publish hard-fails if any single file is 1 MB or larger.
- Disparity-family values (D and its CIs and differences, worst-case gap, per-concept D, heatmap |Δs|) and consistency scores are on the 0–100 presentation scale; rates (repeat instability, contested rate) are fractions in [0, 1]; receipt group_summary s-values stay on [0, 1].
- Every number that carries uncertainty is published as {value, ci: [lo, hi]} (95% percentile cluster-bootstrap CIs, B = 2000, seeded).
- Receipt ids are <bank_version>:<concept_id>:<model_file_slug>; model file slugs are the model slug with / → __ and : → --.
See Reproduction for how a run is regenerated from a pinned bank.
Bank-version changelog
| Bank | Date | Notes |
|---|---|---|
| v1.0.0 | 2026-08-06 | Frozen after pilot screening. No items were dropped: no concept hit the 80% universal-refusal prune and no rubric fell below the κ gate. The freeze changes the bank hash, so v1.0.0 scores are not comparable to v0.1.0 scores. |
| v0.1.0 | 2026-08 | Pre-release pilot bank: 7 demographic attributes × 16 concepts plus 2 control attributes × 10 concepts. Used by the pilot run (and the earlier synthetic sample run that exercised the schema). |
Any edit to any template, group list, or rubric changes the bank hash and starts a new comparison universe; runs are comparable only within a bank hash. Bank versioning and hashing