About AIBiasBench
AIBiasBench is a NIST-aligned benchmark measuring consistency of treatment in large language models. It asks each model matched pairs of prompts that are identical except for a single group term and measures whether the answer changes. It is a measurement instrument, not a scoreboard: magnitude of disparity is the headline, and direction is published only as a clearly labeled diagnostic.
A matched pair: identical except the group term
What it measures, and why consistency first
The design generalizes the paired-prompt idea used by earlier single-axis bias benchmarks: swap one demographic term, hold everything else fixed, compare the answers. AIBiasBench extends it in three deliberate ways:
- All pairs, no privileged baseline. Earlier counterfactual benchmarks typically compared each group against a designated reference group. Choosing that reference is a normative act: it defines whose treatment counts as "normal". Here disparity is dispersion across all groups simultaneously (the Gini mean difference): every pair counts symmetrically, and a flip toward or against any group counts equally. Why no baseline
- Consistency as the headline. The score answers one falsifiable question (did the answer change when only the group term changed?) rather than a contested one (which direction of difference is bad, and for whom). A model that treats every group identically scores 100 regardless of what its answers were.
- Direction as diagnostic. Which groups received more favorable treatment is still published, as directional lean, quarantined from the headline score, multiplicity-corrected, and explicitly labeled as neither a ranking nor a claim about intent.
Demographic scores are never read against a perfect 100 but against each model's own control band: the noise its output shows under non-demographic substitutions (professions, arbitrary tags). The benchmark makes no demographic claim beyond what that noise allows.
Who runs it
AIBiasBench is an independent, single-maintainer project: prompt-bank authoring, the evaluation harness, the judging protocol, and this site. Evaluation costs are paid out of pocket; the project accepts no model-vendor funding. No evaluated vendor has any input into the bank, the metrics, or publication decisions.
Judging disclosure
The current run (full-2026-08) declares two primary judges (openai/gpt-5-mini and google/gemini-3.1-flash-lite) with anthropic/claude-haiku-4.5 as tiebreak, but the panel is chosen per subject model, not fixed: a judge never scores a model from its own vendor family, so same-vendor judges are rotated out and the effective pair is recorded per model in the manifest and per row in the receipts.
That rotation also consumes the tiebreak judge. For 7 of the 14 models in this run the tiebreak judge is either promoted into the primary pair or rotated out for sharing the subject's vendor, leaving no third vote at all (anthropic/claude-opus-5, anthropic/claude-fable-5, anthropic/claude-sonnet-5, openai/gpt-5.6-sol, openai/gpt-5.6-terra, google/gemini-3.1-pro-preview, google/gemini-3.6-flash). For those models any disagreement between the two primary judges is recorded as contested immediately, so their contested rates are not comparable with the 7 models that do get a third vote.
All judges see group terms masked to [PERSON] in both prompt and response. Inter-judge agreement on this run is Cohen's κ 0.57–1.00 by rubric (gate: κ ≥ 0.70, or PABAK ≥ 0.70 when one label exceeds 90% prevalence; amended 2026-08-07), with a contested rate of 3.4%; contested rows are excluded from scores, never coerced. Per-rubric and per-subject-vendor κ are published in the run manifest. Full judging protocol
Who runs this
AIBiasBench is built and operated by Ross Chambers, a third-party risk management and AI implementation consultant based in the Greater Toronto Area. He runs Arc North Services Inc., an independent practice focused on enterprise third-party risk programs, ServiceNow vendor risk implementations, and the practical application of AI in governance, risk, and compliance work. Before going independent he spent four years at Wipro, first as a cyber security consultant and then as a senior consultant leading third-party risk transformations for enterprise clients. He studied computer science at Sheridan College and holds the Certified in Cybersecurity credential from (ISC)².
Assessing vendor risk for a living shapes how this site reports: versioned inputs, published evidence, uncertainty on every number, and documented limitations. Questions, disputed receipts, and methodology critiques are welcome via LinkedIn.
Independence and funding
Independent project; evaluation costs paid out of pocket; no model-vendor funding.
Verification without source access
The evaluation harness and prompt bank are not open source. The published artifacts are designed to make that matter as little as possible: every prompt, every model response, every judge vote summary, and every score is downloadable from the data page and browsable in the receipts browser, and the methodology specifies the pipeline, formulas, and judging protocol in enough detail to reimplement and check the published numbers. A disputed receipt is treated as a first-class bug report.
FAQ
- Why tiers instead of ranks?
- Because a rank implies a distinction the data may not support. When two models' 95% confidence intervals overlap, their ordering is noise; publishing it as a ranking would manufacture a difference. Models are therefore grouped into tiers by CI overlap, and the raw rank appears only in small print. How tiers are computed
- Why does no model reach 100?
- Because no model is perfectly deterministic under substitution. Swapping any term ("an accountant" for "a barista", one favorite color for another) perturbs output slightly, and provider-side nondeterminism adds noise even at temperature 0. That baseline is measured per model as the control band; a demographic score at or inside the band is at the instrument's effective ceiling. 100 would require byte-identical treatment of every group on every concept, which substitution noise alone rules out.
- Why are refusals not scored as stances?
- A refusal is a different behavior from an answer: it says the model would not engage, not what the model believes. Coding it as "NO" (or any value) would let safety-layer asymmetries masquerade as stance disparities, or dilute real ones. Refused rows carry no value and feed a separate metric, refusal asymmetry, which measures engagement asymmetry directly.
- Why small judges rather than a frontier model?
- Every response is classified by two primary judges, with a tiebreak judge behind them wherever the cross-vendor rotation leaves one free; for the pilot run that was more than twice as many judge calls as model calls. Masked verdict classification with strict labels is a narrow task that small judges handle well, and the protocol gates on it: inter-judge agreement must reach κ ≥ 0.70 per rubric (or PABAK ≥ 0.70 where one label exceeds 90% prevalence and deflates κ; see the amended gate under Judging) or the category is re-judged under a revised rubric. The choice is also testable directly: the harness includes a frontier-judge spot-check (tools/frontier_judge_check.py) that re-labels a stratified sample of judged rows with a single frontier judge under the identical masking and output schema and reports agreement with the panel's final labels. Its results are not yet part of the published run artifacts, so no agreement figure is claimed here until they are. What is published with every run: judge names, per-rubric and per-vendor κ, and the contested rate.
What a score means
A high consistency score means uniform treatment on this prompt bank under this bank version, nothing more. It does not certify a model as unbiased, does not measure real-world harm, and does not evaluate intent. See limitations and the methodology.