How to Evaluate an Analytics Agent

A data team turned one of its most complex dashboards into an analytics agent. It took two months to build. Five months later, it is still not in production, held back by source data from two unreconciled systems and validation rules skipped in the rush to ship.

The stakeholder’s verdict was blunt: “I still cannot trust the data. There’s no difference between my dashboard and this agent. They still make the same mistake.” The fix meant building a whole domain’s worth of data products, so every answer considered the right data first.

The technology had moved on. The question it left unresolved had not: how can I trust this number? Grounding agents in governed data products makes them more accurate, but far less is said about the step after: validating it worked before the business validates it for you.

How Do You Evaluate an Analytics Agent?

Evaluate an analytics agent against a question set that grows from real user questions. Score its results deterministically against expected answers written by the domain owner, re-run the set whenever anything underneath the agent changes, and certify the agent one domain at a time.

Why Do Analytics Agents Fail Silently?

A broken dashboard usually looks broken, with a blank tile or a spike to zero. A wrong agent answer looks exactly like a right one. The SQL runs, the number is plausible, and the explanation is fluent, so silent wrong answers erode trust faster than any visible error. Give the agent three revenue tables with conflicting numbers, and it will pick one without telling anyone.

None of these is caught by asking whether the query ran, and most slip past golden questions written from existing dashboards, which hard-code the joins, grains, and filters the agent now chooses for itself.

In dbt Labs’ 2026 benchmark, a well-modeled semantic layer approached 100% on the questions it covered but could not answer the rest, while raw text-to-SQL scored 64.5% across the full set. A semantic layer failure looks like an error message. A text-to-SQL failure looks like an answer.

Even rigorous teams say it is unsolved. Anthropic’s data team reports automating 95% of its business analytics queries at about 95% accuracy (self-reported), and concedes the plausible wrong answer is the failure its safeguards do not fully catch.

Why Aren’t Golden Questions Enough?

The recipe is familiar: pick a frontier model, point it at the warehouse or a semantic layer, and write golden questions to tick the accuracy box. Those questions are usually written by the builders, drawn from the dashboards being replaced, and run once before launch. They prove the agent can answer what its builders expected, not what the business will ask.

Public benchmarks do not close that gap. ESQ-Bench, a 2026 enterprise benchmark, notes that models report execution accuracy above 89% on Spider and BIRD, then measures GPT-4o at 79.8%, 60.3%, and 57.2% across three tiers of enterprise schema complexity.

Researchers at Thumbtack make the same point: grading only the final answer hides where analytics agents fail, whether through the wrong source of truth, a skipped step, or a table read differently from one run to the next.

How Do You Build an Evaluation Set for an Analytics Agent?

Treat the question set as a product: owned, versioned, and growing. It rests on three decisions: where the questions come from, which kinds to include, and who writes the expected answers.

Where Do the Questions Come From?

Seed, mine, expand: three steps to an analytics agent evaluation set.

• Seed: five to ten questions from the developer, the classic golden questions.
• Mine: reverse the SQL already run against the warehouse into the business questions it answered.
• Expand: collect at least five real questions per user, in their own words, then keep sampling the agent’s logs after launch.

Mining produces evaluation questions, not training material for the agent. Anthropic’s data team found that giving its agent raw access to thousands of historical queries moved accuracy by less than a point. So distill history into questions instead of handing it over raw, as a means to prioritise diversity and quality over quantity. Dumping an entire set without human scoping can lead to innacuracy.

Which Kinds of Questions Belong in the Set?

Sourcing is half the job. The set also needs the right mix:

• Answerable: the data and definitions exist, and there is one correct number.
• Ambiguous: “How are sales doing?” The right behavior is a clarifying question, not a confident guess.
• Unanswerable: the data is missing, or the user lacks access. The right behavior is to say so plainly.

An agent that scores 95% on answerable questions but invents numbers for the rest is more dangerous than one that scores 85% and knows its limits.

Who Writes the Expected Answers?

The domain does. The person who knows what Q3 means in each business unit is not the AI engineer. A data product owner, who sits in the domain and bridges the data team and its stakeholders, is the natural author of the standard every answer is held to. Tag each question with its domain, difficulty, and cost of error.

What Does a Correct Answer Mean?

Most teams measure one thing: did the agent produce a query that ran? That is the lowest rung of a three-rung ladder.

1. It ran. The query executed without error. Necessary, but nearly meaningless alone.
2. It matched. The result equals the expected result within an agreed tolerance for rounding and ordering.
3. It was right for the business. The number matched, used the correct definition, stated its assumptions (“I used fiscal Q3”), and flagged caveats.

Notice what is missing: comparing the agent’s SQL to a reference query. Two queries can look nothing alike and return the same correct result. ESQ-Bench found execution match between roughly 57% and 80% on enterprise schemas while exact SQL match stayed below 7%. Compare results, not SQL text.

Then decide who does the checking. Numbers are checked deterministically, and words are judged. A script can confirm whether the agent returned 4.2 million. A model judge earns its place on whether the explanation was clear and honest about assumptions, calibrated against a human on a sample, because research on LLM judges documents position, verbosity, and self-enhancement biases alongside its 80% agreement with humans.

BADGER, a Merkle evaluation framework, applies the same split: a model aligns the structure of two result sets, and scoring is then deterministic, cell by cell. On 150 human-annotated industry queries, it reached 87.3% balanced accuracy and substantial agreement with human experts (Cohen’s kappa of 0.717). Even so, the best automated grader still disagrees with experts on a meaningful share of answers, so the domain owner stays in the loop on a sample.

Finally, check the path as well as the destination. An agent that needed six failed queries to reach the right answer got lucky. BADGER’s Excess Tool Usage metric counts steps beyond what the task required, and the Thumbtack study found one configuration changed its table interpretation on 41 of 50 questions.

How Often Should You Re-Run an Evaluation?

Accuracy belongs to the agent, and everything around it, and all of it keeps moving. Offline accuracy can fall from roughly 95% at launch to about 65% within a month. Anthropic observed this decline before treating reference-document upkeep as an engineering problem. The model did not get worse. The data model changed, and the documentation went stale.

Re-run the evaluation set whenever any of these change:

• A metric definition changes in the semantic model or data product.
• A source schema changes, or a table is deprecated.
• A new data product becomes visible to the agent.
• The model or its version changes, including silent provider upgrades.
• The agent’s instructions or tools are edited.

These are the events that already trigger tests on the data product, a layer covered in Modern Data 101’s Data Product Testing Strategy handbook. Put the agent evaluation in the same pipeline, one layer up: product tests ask whether the data is right, and agent evaluations ask whether the answer built on it is. A quality alert that arrives after an agent has acted is observability, not control, so both should fail before the agent does.

In production, sample a handful of real questions weekly, have the domain owner grade them against the ladder, and promote every miss into the set.

Who Signs Off on an Analytics Agent?

“Is the agent accurate?” is the wrong question, because no agent is accurate everywhere. Ask for which domains and kinds of questions it has earned trust. An agent is not launched. It is certified, domain by domain, and the agent should say so when a question crosses that boundary.

Anthropic’s data team gates its agent this way: a domain owner cannot announce it to stakeholders until that domain’s slice of the evaluation set clears a threshold, initially about 90%. The sign-off belongs with the product owner who wrote the expected answers and carries the accountability when a wrong number reaches a decision, not with the team that built the agent.

This also changes how time-to-market is read: the measure is how long until a domain is certified, then how long the next question takes, which should approach zero. The first domain takes about as long as a serious dashboard did, and the second is faster because the harness and sign-off process are reused.

How DataOS Enables High-Performing Analytics Agents

Evaluation tells you whether an answer is right. It does not make the underlying data right, and that is where DataOS works. Every data product carries its own semantic model, so the join and definition choices behind the silent failures above are made once in the model, not guessed again by the agent on every question.

Data contracts set SLAs for quality, freshness, and delivery, and DataOS continuously validates them, so a stale source fails visibly. Lineage and contracts travel with each data product, so a changed metric or deprecated table is visible at the source, where the trigger list above begins.

DataOS does not write the expected answers or sign off a domain. Those stay with the people who own the metric. It makes sure a failed evaluation points to a fixable layer, not a prompt.

How to Certify Your First Domain

Pick one domain where the definitions are settled. Seed the set, mine the query history, collect five real questions per user, and have the domain owner write the expected answers, including for ambiguous and unanswerable questions. Set the threshold before the first run, and let the agent go live for that domain only when it clears it.

Dashboards did not die because they were slow. They died because, one erroneous tile at a time, people stopped believing them. A confident wrong answer travels faster than a broken chart ever did, so the real work is earning the right to say “you can trust this number.”

‍