Evaluating an Analytics Agent

Evaluating an analytics agent (or team of agents) is WAY harder than evaluating a chatbot, harder than evaluating a search system, and considerably harder than evaluating a regular ETL pipeline. The reason: the right answer isn’t just a string or a document ranking; it’s a number that has to be exactly right, computed from data that’s constantly changing, by a system that takes a different path through your data on every run. Worth taking seriously before you ship one of these to a CFO.

What Evaluating Has To Cover

An analytics agent takes a business question and returns a number with an explanation. The honest evaluation has to cover several things, each of which has its own failure modes.

• Correctness. Is the number right?
• Faithfulness. Does the explanation match what the system actually did?
• Refusal. When the system can’t answer, does it say so?
• Consistency. Does the same question produce the same answer on different runs?
• Robustness. Does the system handle ambiguity, misspellings, or partial questions?
• Cost. What does each question cost in tokens, tool calls, and compute?
• Latency. How long does it take to answer?

The Ground Truth Problem

Correctness needs a ground truth set: questions with known answers. Building this is the most underestimated piece of work in agent deployment. A good set covers the distribution of real questions, includes ambiguous ones, includes ones the system should refuse, and is maintained as the underlying data changes.

Teams that take this seriously build a frozen test corpus (versioned data, fixed answers) for regression testing, and a live monitoring set (questions whose answers are recomputed daily) for drift detection.

What Correctness Actually Means for a Number

It’s subtler than you’d think. The agent says “revenue was £4.2 million last quarter.” The ground truth says “£4,184,523.” Is that right? Depends what you call right. The rounding is reasonable. The currency conversion might be off. The agent might have included or excluded refunds. The agent might have used a different time zone for “quarter.” All of these can produce a number that’s nearly right but not authoritative.

Evaluation has to score this carefully. Exact match is too strict. Within-X% match is more useful, but you need to define X for each metric type. And the explanation has to match the choice the agent made – if the agent excluded refunds, that needs to be in the answer.

Faithfulness

Faithfulness is whether the explanation matches the actual computation. A faithfully-described wrong answer is more useful than an unfaithfully-described right one, because at least you know what went wrong.

The way to evaluate this: capture the agent’s tool calls, the actual SQL or semantic queries it ran, and compare to the natural-language explanation it gave. Do the joins it claims to have done match what the queries actually did? Does the filter it mentions match the WHERE clause? An LLM-as-judge can do this, but for high-stakes deployments you want human spot-checks too.

Refusal

The most dangerous failure mode of an analytics agent is the confident wrong answer to an unanswerable question. Your test set needs questions the system should refuse: questions about data it doesn’t have, time periods outside the data’s coverage, calculations that don’t make sense, contradictions in the question itself.

Refusal rate matters as much as accuracy. A system that answers 100% of questions with 80% accuracy is worse than one that answers 80% of questions with 95% accuracy. The 20% it refused on were the ones it would have got wrong; surfacing the refusal lets the user know.

The Structure of a Working Eval

What I’ve seen work in production:

1. A ground truth set of ~200 questions, curated by domain experts, with known answers, expected refusals flagged.
2. An automated harness that runs the agent against every question on every change to the system.
3. LLM-as-judge for explanation faithfulness, supplemented by human review of a sample.
4. Cost and latency tracking per question.
5. A drift monitor that re-runs answer-changing questions daily and alerts on unexpected shifts.
6. A regression dashboard showing accuracy, refusal rate, cost, latency over time.

None of this is novel research. It’s engineering discipline applied to a new abstraction. The teams that ship working analytics agents are the ones that built this eval harness before they shipped, not after the first executive complaint.

The Deeper Point

Evaluating an analytics agent is hard enough that most teams skip it. The ones that skip it are also the ones that don’t know whether their system works. That’s the same place a lot of organisations were with their data quality programmes pre-2015: certain it must be working, no evidence either way.

The cost of a wrong number in a CFO’s deck is much higher than the cost of building the eval harness. The teams that take this seriously will have systems that justify the trust. The ones that don’t are quietly betting that their agent is right.