AI Operator Evaluation with Token Telemetry
Model evaluation measures the AI model itself. Operator evaluation measures the operator using the AI. SigRank evaluates who ranks highest at driving their AI under the stated metric and time window, built on real token telemetry, not preference votes.
What is an AI operator?
An operator is the account associated with AI-work telemetry. It is not necessarily a legal person, employer, or unique human: one operator can have devices, submissions, and an optional authenticated account link. In practice, an operator is the entity whose token usage is measured — the account driving the AI tool, not the AI model itself.
The term has multiple meanings in the wild. OpenAI's Operator product is an autonomous agent. In business, an "AI operator" can be a job title. In SigRank's measurement layer, an operator is the account whose token telemetry is collected and scored. This is a measurement definition, not a job-title definition.
AI operator vs AI agent vs AI model
| Concept | Role |
|---|---|
| Operator | The individual whose AI-work telemetry is measured — the unit of analysis for measurement. |
| Agent | Software executing tasks autonomously. Distinct from the operator — the agent is a tool, not the measured subject. |
| Model | The underlying inference system (GPT, Claude, Gemini). Model leaderboards rank models; operator scoring ranks operators. |
| Operator-system dyad | The operator and AI system measured together. Telemetry captures the dyad's behavior, not the operator in isolation. |
| SigArena | The public leaderboard surface (sigeconomy.com). Displays rankings. |
| SignalAF | The public distribution and platform brand (signalaf.com). Hosts scoring, methodology, and enrollment. |
| Upsilon | The SignalAF measurement engine. Consumes telemetry, produces diagnostic measures. Implements TTEOP. |
| TTEOP | The open, vendor-neutral interoperability protocol for token telemetry measurement. Defines primitives, metrics, and conformance. |
What telemetry can observe — and what it cannot prove
Operator telemetry observes token-flow patterns: how much fresh input an operator sends, how much output the system produces, and how much cached context is read or written. From these four primitives, Upsilon computes Yield and related metrics.
Token-flow metrics do not measure productivity, work quality, professional skill, task correctness, cognition, employee performance, or business value. A high Yield means an operator ranks highly under the stated metric and time window — nothing more. Conflating token-cascade efficiency with any of these is a category error.
Get your operator score
Install the SigRank CLI, enroll your operator identity, and submit a signed token-telemetry snapshot. See your Yield, your class tier, and your rank on the public leaderboard. Token counts only — never prompt content, never code.
Evaluating a team, not just yourself?
The public leaderboard ranks individual operators. If you need a private evaluation for a team or workflow — cohort-level analysis, intervention testing, and a defensible baseline report — MO§ES™ offers a 30-day baseline assessment using the same content-free telemetry.
Explore the baseline assessment →Relationship to the ecosystem
What operator scoring is
AI operator scoring is the systematic ranking of operators — the accounts associated with AI-work telemetry — by the efficiency of their token cascade. It does not rank which model is smartest. It does not rank who spent the most hours at the keyboard. It ranks who ranks highest at operating their AI under the stated metric and time window — who compounds signal, who reuses context, who turns a small fresh input into dense output. The unit of measurement is the token, and the architecture of the cascade is the score.
It is a new performance layer because it sits between two existing layers that each miss half the picture. Model benchmarks rank the AI but ignore the operator. Time trackers rank the operator but measure the wrong currency — hours, not tokens. Operator scoring ranks the operator in the currency that actually matters for LLM compute: token-cascade efficiency.
How it differs from model benchmarking
Model benchmarking asks which AI is best? and ranks models by test-suite scores (MMLU, HumanEval, SWE-bench) or preference votes (LMSYS Chatbot Arena). It holds the operator as a constant — the same harness, the same prompts, the same evaluators — and varies the model. Operator scoring inverts that: it holds the model as a constant (you pick one and drive it) and varies the operator. The data is not a synthetic test or a vote; it is real token telemetry from live coding sessions, captured on-device. The question it answers is not “which AI is best?” but “who ranks highest at operating the AI under the stated metric and time window?”
The two are complements. A model benchmark helps you choose a model. Operator scoring helps you measure whether you are driving the model you chose well. Most developers have never had the second measurement. That is the gap SigRank fills.
How it differs from time tracking
Time trackers — WakaTime and its peers — measure hours spent coding. They reward presence: the more time you log, the more productive you appear. But time is a proxy for productivity, not a measure of it. A developer who compounds signal in two hours outscores one who burns tokens for eight — but the time tracker ranks them in the opposite order. Operator scoring measures the actual currency of LLM compute: tokens. It rewards the architecture of the cascade, not the duration of the session.
This matters more as AI coding becomes the dominant mode of development. When the bottleneck is not how fast you type but how well you drive an AI, hours become a misleading metric. Token efficiency is the number that tracks the skill that actually matters.
The SigRank scoring system
Scoring starts with four on-device token pillars: input, output, cache-read, cache-write. The yield metric Υ = cache_read × output / input² is the headline — it measures the architecture of the cascade in one number. The composite SIGNA rate blends yield with signal-force and drift components to produce the final operator score. Proprietary weights (RS.xx) govern the composite and remain server-side — the scoring shape is public, the exact weights are not, to prevent gaming.
Operators are classified into 8 experience tiers - IGNITER through ARCH+ - and ranked over 7-day, 30-day, 90-day, and all-time windows. Snapshots are ed25519-signed on-device and verified server-side with replay and plausibility guards. The system is platform-neutral: it works across Claude, ChatGPT, Gemini, Copilot, Cursor, and 15+ platforms. The foundation is a published conservation law for language under compression (DOI: 10.5281/zenodo.20029607).
Privacy preservation
Operator scoring is privacy-preserving by construction. The on-device scanner reads token counts only — four integers per session. It never reads the content of your prompts or the model's responses. Only ed25519-signed numeric scores leave your device. Server-side verification operates on integers with replay and plausibility guards, never on text. Your prompts, your code, and your conversation history never leave your machine.
This is what makes a global, continuous operator leaderboard possible. You cannot publish a ranking built on prompt content without a privacy scandal. You can publish one built on four signed integers. Token counts are the unit that makes operator scoring safe enough to run at scale.
Explore the category
Operator Performance — Scoring the Operator
The companion hub: why the operator is the variable, how SigRank scores operators, and the class tiers from IGNITER to ARCH+.
The SigRank Index — Methodology
The canonical methodology: how operator scores are computed from token telemetry, how snapshots are verified, and how the leaderboard is ranked.
The Conservation Law of Commitment
The academic foundation: a published conservation law for language under compression, with Zenodo DOIs and an empirical record.
Yield (Υ) Cascade
The headline metric in the scoring system: cache_read × output / input². The architecture of the cascade, not raw spend.
SigRank vs. LMSYS Chatbot Arena
LMSYS Arena ranks AI models by human preference. SigRank ranks operators by token-cascade efficiency. Different questions, different answers — and why both matter.
The SIGNA Score
The composite operator score that blends yield with signal-force and drift. How the number on the leaderboard is computed and what it means.
FAQ
- What is AI operator scoring?
- The systematic ranking of operators — the accounts associated with AI-work telemetry — by token-cascade efficiency. It ranks who ranks highest at operating their AI under the stated metric and time window — measured by yield and the composite SIGNA rate — not which model is best or who spent the most hours.
- How does it differ from model benchmarking?
- Model benchmarking ranks models by test scores or votes — it asks “which AI is best?” Operator scoring ranks operators by real token telemetry — it asks “who ranks highest at operating the AI under the stated metric and time window?” One holds the operator constant; the other holds the model constant. They are complements.
- How does it differ from time tracking?
- Time tracking measures hours — it rewards presence, not efficiency. Operator scoring measures token-cascade efficiency — it rewards the architecture of how you use the AI. A developer who compounds signal in two hours outscores one who burns tokens for eight.
- What is the SigRank scoring system?
- Four on-device token pillars feed yield (Υ = cache_read × output / input²) and the composite SIGNA rate. Weights (RS.xx) are server-side. Operators are tiered (IGNITER to ARCH+) and ranked over multiple windows. Snapshots are ed25519-signed and verified.
- Is operator scoring private?
- Yes. The scanner reads token counts only — four integers, never prompt content. Only ed25519-signed numeric scores leave your device. Server-side verification operates on integers, not text. Your prompts, code, and conversation history never leave your machine.