AI Agent Evaluation — Measuring the Human Directing the Agent
AI agent evaluation focuses on the agent — task completion, tool use, latency. But agents are directed by humans. SigRank measures the operator directing the agent — the variable that determines whether the agent succeeds.
What agent evaluation measures today
Agent evaluation today focuses on the agent itself. Task completion rate: did the agent finish the task? Tool-use accuracy: did the agent call the right tools in the right order? Latency: how long did the agent take? Cost per task: how many tokens did the agent burn? These are necessary metrics — but they measure the agent, not the human directing it.
The blind spot is the operator. The operator determines the context the agent works in, the prompts that frame each task, and the workflow that connects tasks. Two operators directing the same agent produce wildly different results — one may complete 90% of tasks efficiently, the other 40% while burning three times the tokens. Agent-only evaluation cannot see that difference because it holds the operator as a constant and averages it away.
The operator is the variable
In practice, the agent is a constant and the operator is the variable. You pick an agent and deploy it; the question that remains is whether your team is directing it well. That question has been unanswerable until now — not because it is unimportant, but because there was no privacy-preserving way to measure it. Reading prompts is invasive; counting tokens is not.
SigRank closes the gap. Four token pillars — input, output, cache-read, cache-write — are captured on-device from real agent-directed sessions. The yield metric Υ = cache_read × output / input² measures whether the operator's cascade is compounding signal or burning tokens. Snapshots are ed25519-signed and verified server-side. No prompt content is ever read — only token counts.
Complement to agent evaluation, not a competitor
Agent evaluation tools like Braintrust and Langfuse measure the agent: traces, tool calls, task completion, latency. SigRank measures the operator directing the agent. Both layers matter. An agent driven poorly still produces poor results; an agent driven well can outperform a stronger agent driven poorly. Complete agent evaluation needs both the agent layer and the operator layer — and the operator layer is the one most teams are missing.
Explore the category
AI Evaluation — Measuring the Operator, Not Just the Model
The four-layer model of AI evaluation: model, output, safety, operator. Agent evaluation sits at the operator layer — measuring the human directing the agent.
The SigRank Index — Methodology
How operator scores are computed from four token pillars, verified server-side, and ranked. The canonical methodology for evaluating agent operators.
SigRank vs. Braintrust
Braintrust evaluates LLM applications and agent outputs. SigRank evaluates the operator directing the agent. Different layers, different questions.
SigRank vs. Langfuse
Langfuse traces LLM application behavior. SigRank measures the human driving the LLM. Tracing the agent vs. evaluating the operator.
FAQ
- What is AI agent evaluation?
- The measurement and comparison of AI agent performance — task completion, tool use, latency, cost. Most agent evaluation measures the agent. SigRank measures the operator directing the agent.
- Why evaluate the operator, not just the agent?
- The operator is the variable. You deploy an agent; the question that remains is whether your team directs it well. Agent-only evaluation averages operator difference away. SigRank makes it visible, continuously, without reading prompts.
- How does SigRank evaluate agent operators?
- Four token pillars captured on-device from real agent-directed sessions. Yield (Υ = cache_read × output / input²) measures cascade architecture. ed25519-signed snapshots, cohort-relative ranking. No prompt content ever read.
- Is SigRank a replacement for agent evaluation tools?
- No — it is a complement. Agent tools (Braintrust, Langfuse) measure the agent. SigRank measures the operator directing the agent. Complete agent evaluation needs both layers.