vyakti-judge-qual
The full judge-qualification protocol, harness and dataset behind the paper “It's Not the Code-Switching”. Released so other teams evaluating LLM judges on affective or open-ended preference tasks can qualify their own candidates against a real backtest rather than assuming trustworthiness.
Built and internally gated; public repository URL pending alongside the paper's arXiv posting.
Repository URL posts with the arXiv preprint
What this data is not.
The datasheet states its limits in its own first section, so this page does too.
- The ground truth was produced by an AI model, not by human annotators. Every agreement figure in this release means agreement with one trusted judge, and never accuracy.
- 96 conversation units clustered on 12 beats. That is adequate to reject an 80% bar and inadequate for finely ranking the failures against each other.
- One product, one persona and one language pair, on two archives. Generalisation is an open question here, never a claim.
- Judge results are date-stamped evidence, 15 and 18 August 2026, rather than properties of a model name. A deployment shifting behaviour over days has already been measured on this programme.
- Adopting this as a benchmark of judge correctness is the principal misuse risk. It measures agreement with one trusted judge's decisions on one product's construct.
In the release
- The generalised qualification harness, including the transport-validity and parse-validity guards that make a run self-invalidate rather than report a crippled result. The paper describes these as a contribution in their own right, not as incidental plumbing.
- 192 ground-truth verdicts with free-text rationales, across seven judged axes (warmth, humour, register, specificity, brevity, personhood, overall). The scarcest asset in the release.
- Both arms of both source archives, stripped to {user, reply} plus {model, lane, beat, replicate}. No persona prompt text, no production user data.
- 1,536 primary judgment rows plus 5,760 six-axis extension rows and 960 English-translation-control rows: every judgment call any figure or table in the paper is computed from.
- A per-vendor deployment quirk log (token-parameter names, silent hidden-reasoning token burn, output-format violations) that practitioners are expected to use more than the headline results.
- A Gebru-style datasheet stating, in its own first section, that the ground truth is LLM-produced rather than human-annotated, and that every agreement figure means agreement with one trusted judge, never accuracy.
Not in the release
- The product's persona prompt. It is not released and is not needed for any claim the paper makes. Archived files that embedded it are processed by an extraction script rather than copied, and the built bundle is checked by a 22-gate de-identification sweep before anything ships.
De-identification record
- 22 / 22
- gates passed, run against the built bundle rather than the source tree
- 1
- real leak caught and fixed before anything shipped: a provider error message carrying a full cloud tenant hostname
A sweep that finds nothing proves nothing. This one found something.
How to cite it
Cite the paper the release accompanies. Its citation block, and the honest note about what has and has not been posted, live on the paper page.
docs/paper/CAMERA.md §7; docs/paper/DRAFT.md §13.4