AI Operator Evaluation with Token Telemetry
Model evaluation measures the AI model itself. Operator evaluation measures the operator using the AI. SigRank evaluates who ranks highest at driving their AI under the stated metric and time window, built on real token telemetry, not preference votes.
What is an AI operator?
An operator is the account associated with AI-work telemetry. It is not necessarily a legal person, employer, or unique human: one operator can have devices, submissions, and an optional authenticated account link. In practice, an operator is the entity whose token usage is measured — the account driving the AI tool, not the AI model itself.
The term has multiple meanings in the wild. OpenAI's Operator product is an autonomous agent. In business, an "AI operator" can be a job title. In SigRank's measurement layer, an operator is the account whose token telemetry is collected and scored. This is a measurement definition, not a job-title definition.
AI operator vs AI agent vs AI model
| Concept | Role |
|---|---|
| Operator | The individual whose AI-work telemetry is measured — the unit of analysis for measurement. |
| Agent | Software executing tasks autonomously. Distinct from the operator — the agent is a tool, not the measured subject. |
| Model | The underlying inference system (GPT, Claude, Gemini). Model leaderboards rank models; operator scoring ranks operators. |
| Operator-system dyad | The operator and AI system measured together. Telemetry captures the dyad's behavior, not the operator in isolation. |
| SignalAF | The public distribution and platform brand (signalaf.com). Hosts scoring, methodology, and enrollment. |
| Upsilon | The SignalAF measurement engine. Consumes telemetry, produces diagnostic measures. Implements TTEOP. |
| TTEOP | The open, vendor-neutral interoperability protocol for token telemetry measurement. Defines primitives, metrics, and conformance. |
What telemetry can observe — and what it cannot prove
Operator telemetry observes token-flow patterns: how much fresh input an operator sends, how much output the system produces, and how much cached context is read or written. From these four primitives, Upsilon computes Yield and related metrics.
Token-flow metrics do not measure productivity, work quality, professional skill, task correctness, cognition, employee performance, or business value. A high Yield means an operator ranks highly under the stated metric and time window — nothing more. Conflating token-cascade efficiency with any of these is a category error.
Model effects are not always held constant across operators. Someone using a model with better prompt caching may show higher cache-read counts independent of their own skill. Task complexity, domain familiarity, and workflow structure all influence the cascade shape. Use operator performance metrics as a signal worth investigating, not a verdict on the person.
Get your operator score
Install the SigRank CLI, enroll your operator identity, and submit a signed token-telemetry snapshot. See your Yield, your class tier, and your rank on the public leaderboard. Token counts only — never prompt content, never code.
Evaluating a team, not just yourself?
The public leaderboard ranks individual operators. If you need a private evaluation for a team or workflow — cohort-level analysis, intervention testing, and a defensible baseline report — MO§ES™ offers a 30-day baseline assessment using the same content-free telemetry.
Explore the baseline assessment →Relationship to the ecosystem
What operator scoring is
AI operator scoring is the systematic ranking of operators — the accounts associated with AI-work telemetry — by the efficiency of their token cascade. It does not rank which model is smartest. It does not rank who spent the most hours at the keyboard. It ranks who ranks highest at operating their AI under the stated metric and time window — who compounds signal, who reuses context, who turns a small fresh input into dense output. The unit of measurement is the token, and the architecture of the cascade is the score.
It is a new performance layer because it sits between two existing layers that each miss half the picture. Model benchmarks rank the AI but ignore the operator. Time trackers rank the operator but measure the wrong currency — hours, not tokens. Operator scoring ranks the operator in the currency that actually matters for LLM compute: token-cascade efficiency.
How it differs from model benchmarking
Model benchmarking asks which AI is best? and ranks models by test-suite scores (MMLU, HumanEval, SWE-bench) or preference votes (LMSYS Chatbot Arena). It holds the operator as a constant — the same harness, the same prompts, the same evaluators — and varies the model. Operator scoring inverts that: it holds the model as a constant (you pick one and drive it) and varies the operator. The data is not a synthetic test or a vote; it is real token telemetry from live coding sessions, captured on-device. The question it answers is not “which AI is best?” but “who ranks highest at operating the AI under the stated metric and time window?”
There is a second problem. Model benchmarks rely on synthetic test suites or human preference votes — not on real coding telemetry. A test suite tells you the model can solve a curated problem in a controlled harness. A preference vote tells you a human liked one response better than another. Neither tells you anything about how efficiently a real developer drives the model through a real coding session over a real week. That gap is where operator time, money, and signal are actually won or lost.
The two are complements. A model benchmark helps you choose a model. Operator scoring helps you measure whether you are driving the model you chose well. Most developers have never had the second measurement. That is the gap SigRank fills.
How it differs from time tracking
Time trackers — WakaTime and its peers — measure hours spent coding. They reward presence: the more time you log, the more productive you appear. But time is a proxy for productivity, not a measure of it. A developer who compounds signal in two hours outscores one who burns tokens for eight — but the time tracker ranks them in the opposite order. Operator scoring measures the actual currency of LLM compute: tokens. It rewards the architecture of the cascade, not the duration of the session.
This matters more as AI coding becomes the dominant mode of development. When the bottleneck is not how fast you type but how well you drive an AI, hours become a misleading metric. Token efficiency is the number that tracks the skill that actually matters.
The SigRank scoring system
Scoring starts with four on-device token pillars: input, output, cache-read, cache-write. The yield metric Υ = cache_read × output / input² is the headline — it measures the architecture of the cascade in one number. The composite SIGNA rate blends yield with signal-force and drift components to produce the final operator score. Proprietary weights (RS.xx) govern the composite and remain server-side — the scoring shape is public, the exact weights are not, to prevent gaming.
Operators are classified into 8 experience tiers - IGNITER through ARCH+ - and ranked over 7-day, 30-day, 90-day, and all-time windows. Snapshots are ed25519-signed on-device and verified server-side with replay and plausibility guards. The system is platform-neutral: it works across Claude, ChatGPT, Gemini, Copilot, Cursor, and 15+ platforms. The foundation is a published conservation law for language under compression (DOI: 10.5281/zenodo.20029607).
Privacy preservation
Operator scoring is privacy-preserving by construction. The on-device scanner reads token counts only — four integers per session. It never reads the content of your prompts or the model's responses. Only ed25519-signed numeric scores leave your device. Server-side verification operates on integers with replay and plausibility guards, never on text. Your prompts, your code, and your conversation history never leave your machine.
This is what makes a global, continuous operator leaderboard possible. You cannot publish a ranking built on prompt content without a privacy scandal. You can publish one built on four signed integers. Token counts are the unit that makes operator scoring safe enough to run at scale.
The operator is the variable
Two operators sit down with the same model — the same Claude, the same ChatGPT, the same Gemini — on the same platform, for the same session length. One walks away with a token cascade that compounds: small fresh input riding a large cached foundation into dense output. The other walks away with a cascade that burns: high input, low cache reuse, thin output. Same model. Same clock. Wildly different results.
The difference is not the model. The difference is the operator — how they structure their prompts, how they reuse context, how they manage the cascade across turns. The model is a constant. The operator is the variable. SigRank is built to measure that variable.
This is the inversion that makes SigRank a different category from model leaderboards. LMSYS Arena ranks which model humans prefer. SigRank ranks which operator drives their model best — highest-ranked under the stated metric and time window. The question is not “which AI is smarter?” — it is “who ranks highest at operating the AI they have?”
Operator benchmarking vs. model benchmarking
| Dimension | Model benchmark | Operator benchmark |
|---|---|---|
| Question | Which AI is best? | Who uses the AI best? |
| Ranks | Models | Humans (operators) |
| Data | Test suites / preference votes | Real token telemetry |
| Setting | Controlled harness | Live coding sessions |
| Example | LMSYS Chatbot Arena | SigRank Index |
The two categories are complements, not competitors. A model benchmark helps you choose a model. An operator benchmark helps you measure whether you are driving the model you chose well. Most developers have never had the second one. That is the gap SigRank fills.
Class tiers explained
Every operator is assigned a class tier from the scoring ruleset. Tiers run from low to high — they describe where an operator is in the compounding journey, not a fixed label.
IGNITER
The entry tier. Operators here are lighting the first sparks - high input, low cache reuse, output still finding its footing. Everyone starts here.
BEARER
Quiet accumulation. Token flow is growing but the cascade hasn't compounded yet.
REFINER
Practicing with purpose. Cache reuse is growing, yield is rising but not yet compounding.
SEEKER
Active exploration. The operator is learning to reuse context and reduce fresh input.
BASE
The center of the field. Mid-tier operators with consistent, balanced token flow.
POWER
Above center. The cascade is taking shape - cache reuse is healthy, output is dense, yield is climbing.
ARCH
System builders. Coherent operators with consistent output and deep context reuse.
ARCH+
The deepest field experience. Sustained high-volume cascade across all dimensions.
The four cascade metrics
Yield (Υ)
Start herecache_read × output / input²The headline metric. Measures the architecture of the cascade — whether signal is compounding or tokens are burning. The single number that captures operator efficiency.
Signal-to-Noise Ratio (SNR)
Quality lensoutput / (input + output)Output share of fresh conversational traffic. High SNR means most of your fresh traffic is model output, not input.
Leverage
Compounding lenscache_read / inputHow much cached context amplifies your fresh input. High leverage means a small input is riding on a large cached foundation — the compounding effect.
Velocity
Secondaryoutput / inputOutput tokens per fresh input token. The output-efficiency metric — useful, but it measures output ratio, not cascade efficiency. High output per input without cache reuse is still wasteful.
How they relate
Yield is the composite. It multiplies cache reuse (cache_read) by output and divides by the square of input. That means yield rises when you reuse cache, when you produce more output, and when you send less fresh input — all at once. It is the one number that rewards the full compounding architecture.
Yield is the product of two of the others: leverage (cache reuse per fresh input, cache_read / input) times velocity (output per fresh input, output / input). Leverage says how well you reuse context but nothing about output; velocity says how much output each fresh input produces but cannot see whether you are compounding. SNR is the output share of fresh traffic. Velocity is secondary to Yield.
Which to prioritize
Start with yield. It is the headline metric and the one the leaderboard ranks by. If your yield is low, open the diagnostic lenses: check leverage to see if your cached foundation is too small relative to fresh input, and velocity to see if your output is thin relative to input. SNR tells you how much of your fresh traffic is output. Velocity is secondary to Yield: do not optimize for it at the expense of yield.
Explore the category
AI Benchmarking Tools — Alternatives
The landscape of AI benchmarking and measurement tools: LMSYS Arena, WakaTime, Cursor metrics, Copilot, ccusage — and where SigRank fits.
How to Measure AI Coding Efficiency
The step-by-step guide to computing the metrics from your token telemetry and reading the results.
The SigRank Index — Methodology
The canonical methodology: how operator scores are computed from token telemetry, how snapshots are verified, and how the leaderboard is ranked.
The Conservation Law of Commitment
The academic foundation: a published conservation law for language under compression, with Zenodo DOIs and an empirical record.
Yield (Υ) Cascade
The headline metric in the scoring system: cache_read × output / input². The architecture of the cascade, not raw spend.
SigRank vs. LMSYS Chatbot Arena
LMSYS Arena ranks AI models by human preference. SigRank ranks operators by token-cascade efficiency. Different questions, different answers — and why both matter.
The SIGNA Score
The composite operator score that blends yield with signal-force and drift. How the number on the leaderboard is computed and what it means.
FAQ
- What is AI operator scoring?
- The systematic ranking of operators — the accounts associated with AI-work telemetry — by token-cascade efficiency. It ranks who ranks highest at operating their AI under the stated metric and time window — measured by yield and the composite SIGNA rate — not which model is best or who spent the most hours.
- How does it differ from model benchmarking?
- Model benchmarking ranks models by test scores or votes — it asks “which AI is best?” Operator scoring ranks operators by real token telemetry — it asks “who ranks highest at operating the AI under the stated metric and time window?” One holds the operator constant; the other holds the model constant. They are complements.
- How does it differ from time tracking?
- Time tracking measures hours — it rewards presence, not efficiency. Operator scoring measures token-cascade efficiency — it rewards the architecture of how you use the AI. A developer who compounds signal in two hours outscores one who burns tokens for eight.
- What is the SigRank scoring system?
- Four on-device token pillars feed yield (Υ = cache_read × output / input²) and the composite SIGNA rate. Weights (RS.xx) are server-side. Operators are tiered (IGNITER to ARCH+) and ranked over multiple windows. Snapshots are ed25519-signed and verified.
- Is operator scoring private?
- Yes. The scanner reads token counts only — four integers, never prompt content. Only ed25519-signed numeric scores leave your device. Server-side verification operates on integers, not text. Your prompts, code, and conversation history never leave your machine.