Confirmation Hacking in AI Evaluation
Confirmation hacking is designing evaluations that confirm what you already believe. SigRank avoids it with content-free telemetry — token counts only, no prompt content, no rubric to rig.
What is confirmation hacking?
Confirmation hacking is the practice of designing AI evaluations that confirm what you already believe, rather than evaluations that test what you actually need to know. It happens when the evaluator controls both the test cases and the grading rubric. You select cases that favor a preferred outcome, write rubrics that score that outcome highly, and discard cases that produce inconvenient results. The evaluation appears rigorous but is rigged toward a predetermined conclusion. It is the evaluation equivalent of p-hacking in statistics.
In AI evaluation, confirmation hacking is especially easy because the evaluator often writes the rubric that grades the output. A rubric that rewards “thoroughness” will favor verbose outputs; a rubric that rewards “precision” will favor terse ones. Either way, the rubric encodes the evaluator's prior. The evaluation measures how well the output matches the rubric, not how well the operator actually performed.
How content-free telemetry avoids it
SigRank removes the evaluator's leverage entirely. It measures operators via token counts only — input, output, cache-read, cache-write — and computes the yield metric Υ = cache_read × output / input² from those counts. No prompt content is ever read, so there is no content to bias the evaluation toward a preferred conclusion. The metric is a fixed mathematical function of four numbers, not a judgment about content quality. There is no rubric to rig.
The data is also tamper-resistant. Snapshots are ed25519-signed on-device and verified server-side, so an operator cannot retroactively alter their token counts. The evaluation is determined by the operator's actual token cascade — the real flow of tokens through a real session — not by a test case the evaluator chose or a rubric the evaluator wrote.
Optimizing is not gaming
An operator can improve their Yield by reusing cache, trimming fresh input, and producing substantive output. That is genuine improvement — the operator is actually driving the AI more efficiently. Content-free telemetry makes the evaluation a measure of real behavior, not a measure of how well the output matches the evaluator's rubric. The only way to raise Yield is to change how you drive the AI. That is the point: the evaluation cannot be hacked toward a preferred conclusion because there is no content to bias and no rubric to rig.
Explore the category
The SigRank Index — Methodology
How operator scores are computed from four token pillars, verified server-side, and ranked. The content-free methodology that avoids confirmation hacking.
The Conservation Law of Commitment
The academic foundation: a published conservation law for language under compression, with Zenodo DOIs and an empirical record. The theory behind content-free evaluation.
AI Evaluation — Measuring the Operator, Not Just the Model
The four-layer model of AI evaluation: model, output, safety, operator. SigRank covers the operator layer with content-free telemetry.
AI Evaluation Frameworks — From Models to Operators
NIST AI RMF, OpenAI Evals, DeepEval, Braintrust, SigRank. How the frameworks compare — and why content-free telemetry matters for avoiding bias.
FAQ
- What is confirmation hacking in AI evaluation?
- Designing evaluations that confirm what you already believe rather than testing what you need to know. The evaluator controls the test cases and the rubric, rigging the result toward a predetermined conclusion. The evaluation equivalent of p-hacking.
- How does SigRank avoid confirmation hacking?
- Content-free telemetry. Token counts only — input, output, cache-read, cache-write. The Yield metric is a fixed function of four numbers. No prompt content is read, so there is no content to bias. ed25519 signatures prevent retroactive alteration.
- Why is content-free telemetry important?
- When evaluation reads prompt content, the evaluator can bias the result by choosing which content to reward. Content-free telemetry removes that leverage. Token counts do not encode opinions about what the operator should have said.
- Can content-free evaluation be gamed?
- It can be optimized for — but that is genuine improvement, not gaming. The metric is fixed and public; the data is signed. The only way to raise Yield is to actually change how you drive the AI. No rubric to rig, no content to bias.