AI Evaluation — Measuring the Operator, Not Just the Model
AI evaluation has four layers: model, output, safety, and operator. The first three are well-served. The operator layer — measuring the humans driving the AI — is the missing piece. SigRank covers it with privacy-preserving token telemetry and the Yield metric.
The four layers of AI evaluation
AI evaluation is not one thing. It is a stack of four distinct questions, each requiring different tools and different data. Most AI evaluation conversation focuses on the first three. The fourth — the operator layer — has been invisible until now.
- 1. Model evaluation. Which model is best? Answered by benchmarks like MMLU, HumanEval, SWE-bench, and preference rankings like LMSYS Chatbot Arena. The most mature layer — and the most saturated.
- 2. Output evaluation. Is the output correct? Answered by human review, LLM-as-judge, test suites, and rubric scoring. Essential for quality assurance, but downstream of the operator — it measures the result, not the driving.
- 3. Safety evaluation. Is the system safe? Answered by red-teaming, alignment benchmarks, and adversarial testing. A compliance and risk layer — necessary, regulated, and orthogonal to who is operating the AI.
- 4. Operator evaluation. Who is best at using the AI? Answered by SigRank. Real token telemetry from live sessions, the Yield metric, and cohort-relative ranking. The layer that determines whether the model you chose is actually being driven well.
Why the operator layer is missing
Model evaluation holds the model as the variable and the operator as a constant. In practice the opposite is true: you pick a model and drive it, so the model is a constant and the operator is the variable. Two operators on the same model produce wildly different results — but model benchmarks average that difference away. The operator layer has been missing because there was no privacy-preserving way to measure it. Reading prompts is invasive; counting tokens is not.
SigRank closes the gap. Four token pillars — input, output, cache-read, cache-write — are captured on-device from real coding sessions. The yield metric Υ = cache_read × output / input² measures whether signal is compounding or tokens are burning. Snapshots are ed25519-signed and verified server-side. No prompt content is ever read — only token counts. It is operator evaluation built on real telemetry, real science, and real privacy.
SigRank's role in the stack
SigRank does not replace model, output, or safety evaluation. It sits beside them as the operator layer — the fourth and previously unmeasured layer of the AI evaluation stack. A team that runs MMLU for model selection, human review for output quality, and red-teaming for safety still has a blind spot: are the developers actually driving the chosen model well? That is the question SigRank answers, with continuous, cohort-relative, governed operator evaluation.
The foundation is a published conservation law for language under compression (DOI: 10.5281/zenodo.20029607), with an empirical record and a public transformation harness. The data is privacy-preserving — token counts only, never prompt content — and cryptographically signed. Operator evaluation is finally measurable without being invasive.
Explore the category
AI Benchmarking — Beyond Model Leaderboards
Model benchmarks rank models. Operator benchmarks rank the humans driving them. The complement to model-only benchmarking that AI evaluation has been missing.
The SigRank Index — Methodology
How operator scores are computed from four token pillars, verified server-side, and ranked. The canonical methodology for the operator layer of AI evaluation.
SigRank vs. LMSYS Chatbot Arena
LMSYS Arena is a model evaluation. SigRank is an operator evaluation. Different layers of the same stack — and why both matter for complete AI evaluation.
Yield (Υ) — Token Cascade Efficiency
The headline operator-evaluation metric: Υ = cache_read × output / input². The single number that captures how efficiently an operator drives the AI.
FAQ
- What is AI evaluation?
- The systematic measurement and comparison of AI system performance across four layers: model, output, safety, and operator. The first three are well-served. SigRank covers the operator layer — the missing piece.
- Model evaluation vs. operator evaluation?
- Model evaluation asks “which AI is best?” and ranks models. Operator evaluation asks “who uses the AI best?” and ranks humans. They are complements — both layers matter for complete AI evaluation.
- How does SigRank evaluate operators?
- Four token pillars captured on-device from real sessions. Yield (Υ = cache_read × output / input²) measures cascade architecture. Operators are ranked, tiered, and scored over multiple time windows. Snapshots are ed25519-signed. No prompt content is ever read.
- Can you evaluate without reading prompts?
- Yes. Token counts are the minimal sufficient statistic for operator evaluation. SigRank captures input, output, cache-read, and cache-write counts only — never prompt content. The data is verifiable via ed25519 signatures without being readable.
- Is operator evaluation a replacement for model evaluation?
- No — it is a complement. Model evaluation helps you choose a model. Operator evaluation helps you measure whether you are driving the model you chose well. Complete AI evaluation needs all four layers.