Why Your Agent Evaluation Metrics Are Lying to You

The dashboard says the agent is improving. The users say it is not.

Both are telling the truth — about different things.

The evaluation metrics measure what the team optimized for: task completion rate, response relevance score, latency, and cost per task. These metrics are green and trending upward.

The users experience something the metrics do not capture: the agent completes tasks but misses context, produces technically correct outputs that require rework, handles the common case well but fails unpredictably on edge cases, and satisfies the evaluation criteria without actually solving the problem the user cared about.

This is Goodhart’s Law applied to agent systems: when the metric becomes the target, it ceases to be a good metric.

Goodhart Effects in Agent Evaluation

Goodhart’s Law — “when a measure becomes a target, it ceases to be a good measure” — operates with particular force in agent systems because:

1. The optimization surface is large: agent systems have many parameters to adjust (prompts, routing, tool selection, output formatting) and will find the cheapest path to satisfying any metric
2. The evaluation surface is narrow: most evaluation sets cover common patterns but not the edge cases where agents fail in production
3. The feedback loop is fast: with automated evaluation, the system can be optimized against the metric quickly, long before human evaluators notice the divergence

The result: the agent learns to satisfy the evaluation metric without improving the quality the metric was supposed to measure.

This pattern appears most clearly when teams run a structured comparison: the same output set, scored by the automated pipeline and then scored independently by a domain expert. When the automated score improves but domain experts still flag substantial rework, the gap is the Goodhart effect in plain view.

Common Goodhart Patterns

Completion gaming: the agent produces outputs that satisfy the structural requirements of “task completed” without actually solving the underlying problem. The completion rate is high, but the rework rate is also high.

Format optimization: the agent learns that well-formatted, confident-sounding outputs score higher with both automated evaluators and human raters — regardless of accuracy. The system optimizes for presentation over substance.

Benchmark overfitting: the evaluation set becomes a training signal. The agent performs well on the benchmark’s question distribution but poorly on real user queries that fall outside that distribution.

Cost-quality tradeoff hiding: reducing tokens per task improves cost metrics but may degrade reasoning quality. The cost metric shows improvement while the quality degradation is invisible because the quality metric does not capture depth of reasoning.

Proxy Collapse

Proxy collapse is what happens when the gap between the evaluation metric and the real-world outcome grows wide enough that the metric becomes meaningless.

Every evaluation metric is a proxy. “Task completion rate” is a proxy for “the user’s problem was solved.” “Relevance score” is a proxy for “the response was actually useful.” “Cost per task” is a proxy for “the system delivers appropriate value for its cost.”

These proxies start useful. They become dangerous when:

• The evaluation set does not represent real production queries
• The evaluation criteria do not capture the failure modes that matter
• The system is optimized against the proxy rather than against the underlying quality dimension
• The team stops checking whether the proxy still correlates with the outcome it was designed to measure

The uncomfortable truth: many teams never check. The proxy was useful at launch. Later, the evaluation set no longer reflects production query distribution, the failure modes the system actually encounters have shifted, and the proxy measures a past version of the problem. The only signal that something changed is the growing distance between what the metrics show and what domain experts see.

from pydantic import BaseModel, Field
from typing import Literal, Optional
from enum import Enum


class DivergenceTrend(str, Enum):
    STABLE = "stable"
    GROWING = "growing"
    SHRINKING = "shrinking"


class ProxyCollapseRisk(str, Enum):
    LOW = "low"
    MEDIUM = "medium"
    HIGH = "high"


class EvaluationDimensionHealth(BaseModel):
    """
    Tracks the health of a single evaluation dimension by comparing
    automated scores against human expert judgment. A growing divergence
    signals proxy collapse — the metric is no longer measuring what it
    was designed to measure.
    """
    dimension: str = Field(description="Named evaluation dimension: accuracy, reasoning, tool_use, format, safety")
    automated_score: float = Field(ge=0.0, le=1.0, description="Score produced by automated evaluation pipeline")
    human_expert_score: float = Field(ge=0.0, le=1.0, description="Score produced by domain expert review on same outputs")
    divergence: float = Field(description="automated_score minus human_expert_score; positive means automated overestimates")
    divergence_trend: DivergenceTrend
    proxy_collapse_risk: ProxyCollapseRisk
    last_validated: str = Field(description="ISO date of last human expert validation run")
    outputs_sampled: int = Field(ge=1, description="Number of outputs reviewed by human expert in last validation")
    recommended_action: Optional[str] = Field(
        default=None,
        description="Engineering action triggered when proxy_collapse_risk is medium or high"
    )

    @property
    def requires_intervention(self) -> bool:
        return self.proxy_collapse_risk in (ProxyCollapseRisk.MEDIUM, ProxyCollapseRisk.HIGH)

Multi-Dimensional Evaluation

The alternative to single-score evaluation is multi-dimensional evaluation that separates the quality dimensions that matter:

Task Accuracy

Did the agent produce a correct answer to the actual question? Not “did the output look good” — did it solve the problem?

This requires ground-truth evaluation data: known-correct answers that the agent’s output can be compared against. Without ground truth, accuracy evaluation is subjective.

Reasoning Quality

Did the agent reason correctly through the problem? This is harder to evaluate than task accuracy because it requires inspecting the reasoning chain, not just the final output.

Evaluation approach: trace the agent’s tool calls, intermediate reasoning, and decision points. Were the right tools called in the right order? Was the reasoning logically sound?

Tool Use Correctness

Did the agent use the right tools with the right parameters? Tool use errors — calling the wrong API, passing incorrect parameters, misinterpreting tool outputs — are a major source of agent failures that single-score evaluation misses entirely.

Output Format and Structure

Does the output match the expected format? This is the easiest dimension to automate and the one most likely to create false confidence — because a well-formatted wrong answer scores high on format evaluation.

Safety and Constraint Adherence

Did the agent stay within its defined boundaries? Did it refuse tasks it should have refused? Did it escalate when it should have escalated?

Safety evaluation is often the most neglected dimension because it requires testing negative cases — situations where the correct behavior is inaction or refusal.

Evaluation Set Drift

The evaluation set is a snapshot of what the team thought mattered when it was created. Production queries evolve. User needs shift. Failure modes change. The evaluation set does not update itself.

Symptoms of evaluation set drift:

• The agent’s production error rate is higher than its evaluation error rate
• User complaints cluster around query types that are not represented in the evaluation set
• The team adds “one-off fixes” to the agent that improve production quality but do not affect evaluation scores

The maintenance practice: refresh the evaluation set on an approved cadence with real production queries, including queries where the agent failed. The evaluation set should reflect what the system actually faces, not what it faced when the set was created.

What Good Agent Evaluation Looks Like

1. Multi-dimensional: separate scores for accuracy, reasoning, tool use, format, and safety
2. Ground-truth anchored: at least one dimension evaluated against known-correct answers
3. Human-validated: automated metrics regularly compared against human expert judgment
4. Production-representative: evaluation set refreshed from real production queries
5. Failure-mode inclusive: evaluation explicitly tests known failure modes, not just the happy path
6. Trend-monitored: divergence between automated and human evaluation tracked over time

Single-score evaluation should not exist in production agent systems. The question is never “what is the agent’s score?” — it is “which dimensions are below threshold, and does any safety dimension have a score that would block release if reported independently?” That framing changes what gets built, what gets reported, and what gets escalated.

For the evaluation layer that should exist before any production AI system scales, see The Evaluation Layer Every Production AI System Needs. For the human feedback design that determines what should block a release, see What Human Feedback Should Block an AI Release. For the observability signals that reveal when metrics are diverging from reality, see What Agent Observability Should Trigger a Production Audit.

• Compare automated evaluation scores against human expert judgment on the same outputs on an approved cadence.
• Separate evaluation into distinct dimensions: accuracy, reasoning, tool use, format, and safety.
• Never aggregate safety scores with other dimensions — report them independently.
• Refresh evaluation sets regularly with real production queries, including failure cases.
• Track the divergence between automated and human evaluation over time — a growing gap signals proxy collapse.
• Test negative cases: situations where the correct agent behavior is refusal or escalation.

FAQ

The Decision Rule

Do not trust a single agent score when users disagree with the dashboard. Split evaluation by task accuracy, reasoning, tool use, format, and safety, then compare automated scores against expert judgment on the same outputs.