Evaluating AI — The Operator Layer Is the Missing Piece
Evaluating AI without evaluating the operator is like evaluating a car without evaluating the driver. You measure the machine but miss the variable that determines real-world performance. SigRank fills the gap.
The car and the driver
Evaluating AI without evaluating the operator is like evaluating a car without evaluating the driver. You can measure the engine's horsepower, the transmission's gear ratios, and the chassis's aerodynamics — and still have no idea who will win the race. The car is a constant; the driver is the variable. The same is true of AI. You can benchmark the model on MMLU, score the output with LLM-as-judge, and red-team the safety — and still not know whether your developers are driving the model well.
Model evaluation holds the model as the variable and the operator as a constant. In practice the opposite is true: you pick a model and drive it, so the model is a constant and the operator is the variable. Two operators on the same model produce wildly different results — token efficiency, output quality, cost per task. Model benchmarks average that difference away. The operator layer is where real-world performance is actually won or lost, and it is the layer that evaluating AI has been missing.
Why the gap existed
The operator layer has been missing for a simple reason: there was no privacy-preserving way to measure it. Measuring the operator means measuring their real workflow — the prompts they send, the context they build, the sessions they run. Reading prompts is invasive and introduces content bias. So the operator layer was left unmeasured, and AI evaluation focused on the layers that could be measured without invading privacy: the model (synthetic tests), the output (scoring the result), and the safety (adversarial testing).
SigRank solves the privacy problem. Four token pillars — input, output, cache-read, cache-write — are the minimal sufficient statistic for operator evaluation, and they can be captured without reading a single prompt. The yield metric Υ = cache_read × output / input² measures whether signal is compounding or tokens are burning. Snapshots are ed25519-signed and verified server-side, so the data is trustworthy without being readable. The gap is closed — not by reading prompts, but by counting tokens.
The complete picture
Complete AI evaluation is a four-layer stack. Model evaluation tells you which AI to choose. Output evaluation tells you whether the result is good. Safety evaluation tells you whether the system is safe. Operator evaluation tells you whether your team is driving the AI well. Each layer answers a different question and needs different data. The operator layer is the missing piece — and with SigRank, it is no longer missing.
Explore the category
AI Evaluation — Measuring the Operator, Not Just the Model
The four-layer model of AI evaluation: model, output, safety, operator. The operator layer is the missing piece of evaluating AI.
AI Benchmarking — Beyond Model Leaderboards
Model benchmarks rank models. Operator benchmarks rank the humans driving them. The complement that evaluating AI has been missing.
The SigRank Index — Methodology
How operator scores are computed from four token pillars, verified server-side, and ranked. The canonical methodology for the operator layer.
The Human in the Loop Is Unmeasured
The blog post that frames the problem: AI evaluation measures everything except the person driving the AI. The operator layer is the blind spot.
FAQ
- What does evaluating AI mean?
- Systematically measuring and comparing AI system performance across four layers: model, output, safety, operator. The operator layer is the missing piece.
- Why is the operator layer missing?
- The model is a constant and the operator is the variable. Model evaluation averages operator difference away. The layer was missing because reading prompts is invasive — but counting tokens is not. SigRank closes the gap.
- How does SigRank fill the gap?
- Four token pillars captured on-device. Yield (Υ = cache_read × output / input²) measures cascade architecture. ed25519-signed snapshots, cohort-relative ranking. No prompt content ever read.
- Is operator evaluation a replacement for model evaluation?
- No — it is a complement. Model evaluation tells you which AI to choose. Operator evaluation tells you whether you are using it well. Complete AI evaluation needs all four layers.