What is the industry standard for evaluating LLM operator performance?

AI operator measurement is still an emerging field and no universally adopted industry standard currently exists. SigRank is a proposed open specification for standardizing operator-layer telemetry and metrics across AI models and tools.

Does SigRank measure productivity?

No. SigRank measures operator-layer telemetry and derived metrics. Productivity, quality, task outcomes, and business value are external dimensions that can be analyzed alongside SigRank measurements.

Does SigRank require reading prompts or source code?

No. The base standard is designed so its core metrics can be computed without requiring prompt text, response text, source code, repository contents, or other semantic payloads.

TTEOP · tteop/0.1-draft

The Open Standard for Operator Metrics

A proposed open measurement specification for the operator layer of generative AI.

Legacy specification

The SigRank Standard (sigrank/0.1-draft) is a legacy specification. The current protocol authority is TTEOP (Token Telemetry Evaluation Operator Protocol, tteop/0.1-draft). The wire protocol identifier sigrank/0.1-draft remains an accepted compatibility alias that resolves to current TTEOP semantics — it is not a second active standard. This page is retained for historical reference and existing consumers. For current protocol semantics, see the methodology page and the tteop-spec repository.

Product boundary

TTEOP, Upsilon, and the wire protocol

TTEOP (Token Telemetry Evaluation Operator Protocol) is the open interoperability protocol authority. Upsilon is SignalAF's commercial measurement engine and the reference implementation for TTEOP. SigRank is the public leaderboard and proof surface for eligible observations. The wire protocol identifier remains tteop/0.1-draft for all records and consumers.

Category definition

Operator performance is a distinct measurement layer

Model benchmarks measure models. Task evals measure task completion. Agent evals measure agent behavior. SigRank defines a portable measurement vocabulary for the operator driving the AI system. It is designed to complement those layers, not replace them.

BUSINESS OUTCOMES
        ↑
ORGANIZATIONAL AI PERFORMANCE
        ↑
AI OPERATOR PERFORMANCE    ← SIGRANK
        ↑
AGENT PERFORMANCE
        ↑
TASK PERFORMANCE
        ↑
MODEL PERFORMANCE
        ↑
INFRASTRUCTURE

Base telemetry

Four primitives

Iinput

Input

Ooutput

Output

Wcache_write

Cache Write / Creation

Rcache_read

Cache Read

Provider and tool aliases are allowed, but a compatible implementation must preserve these primitive semantics when translating them.

Normative v0.1 core

Five operator metrics

Yield (Υ)

(cache_read × output) / input²

Compound relationship between context reuse and output relative to fresh input.

Leverage

cache_read / input

Reusable context amplification relative to fresh input.

Velocity

output / input

Output generated per unit of fresh input.

SNR

output / (input + output)

Output share of the direct input and output exchange.

10xDEV

log₁₀(cache_read / input)

Logarithmic context amplification under the reference implementation policy.

Content independent by design

The base standard does not require prompt text, response text, source code, repository contents, or other semantic payloads to calculate its core measurements.

What SigRank does not measure

SigRank does not inherently determine correctness, task success, code quality, employee productivity, business value, model intelligence, or causal impact. Those can be analyzed as external outcome layers.

Reference architecture

TTEOP

Open interoperability protocol authority.

@sigrank/cascade

Reference math implementation.

SignalAF

Public reference platform and field.

sigrank-mcp

Portable CLI, TUI, and MCP instrument.

Portable operator record

The draft defines a versioned JSON record so tools can exchange the same primitive telemetry and operator metrics without sharing semantic content.

View JSON Schema Current methodology Explore the field

FAQ

What is the industry standard for evaluating LLM operator performance?: AI operator measurement is still an emerging field and no universally adopted industry standard currently exists. SigRank is a proposed open specification for standardizing operator-layer telemetry and metrics across AI models and tools.
Does SigRank measure productivity?: No. SigRank measures operator-layer telemetry and derived metrics. Productivity, quality, task outcomes, and business value are external dimensions that can be analyzed alongside SigRank.
Is this already a universal industry standard?: No. v0.1-draft is a proposed open standard. The objective is to publish an implementable vocabulary, reference implementation, and compatibility path that others can evaluate, critique, and adopt.