You Can't Sample Your Way to Reliable Agents

Vijay Karthik

09 Jul 2026 • 4 min read

Building agents has become easy. A few lines of code and you have a working agent:

from google.adk.agents.llm_agent import Agent

def add(a: int, b: int) -> int:
    return a + b

root_agent = Agent(
    name="simple_math_agent",
    model="gemini-flash-latest",
    instruction="You are a helpful agent. Use the add tool when asked to add numbers.",
    tools=[add],
)

But once it's in production, how do you know it's actually working? Are your customers happy with what the agent is doing?

If you've been in observability, you've seen this before. We used to sample 1% of traces because storing all of them was too expensive. Then someone would have an outage and the trace they needed wasn't in the 1%. The industry fixed it - we store everything now.

With agents, we've walked into the same trap one layer up. We keep all the traces, but evaluating them is the new bottleneck. A single observation might need three or four LLM judges - hallucination, toxicity, helpfulness. Run that on everything and your eval cost exceeds your agent cost. Everyone is back to sampling again, and this time it is evals.

Why LLM-as-a-Judge?

Agents are non-deterministic. You can't just assert on the output like you would with a function. The most common approach is "LLM-as-a-Judge" using another LLM to evaluate the output of your agent.

This works well at small scale, but gets expensive fast. A single LLM observation might need multiple judges - one for hallucination, one for toxicity, one for helpfulness. Run that on every trace and your eval bill exceeds your agent bill!

The standard fix for cost is sampling. Run evals on, say, 1% of your traces.

The problem? The traces you actually care about - the weird edge cases, the frustrated users, the subtle hallucinations - those are rare by definition. Sample 1% and you'll almost certainly miss them.

The Solution: Judge ALL observations

What you really want is to find every interesting observation, not just the interesting ones that happened to land in your sample. Here's how we're approaching this at Oodle.

Store all observations

You can't judge what you don't have. We built a storage engine specifically for LLM traces that stores them at roughly $1 per million observations. That's the starting point - keep everything.

Sentiment Analysis

This one's old school. VADER is a pre-deep-learning NLP technique that scores whether a piece of text has positive or negative sentiment. No LLM calls, no cost per evaluation - just fast, deterministic string analysis.

We run VADER on every LLM observation from the second turn onwards. Why skip the first turn? The agent hasn't done anything yet - the user is just asking a question. Sentiment only becomes meaningful once the agent has responded and the user reacts.

It's not perfect. Sentiment analysis doesn't help when agents do work autonomously without user input, and it's only useful for multi-turn conversations. But for conversational agents, it's a surprisingly effective and free way to flag sessions where the user seems unhappy.

Finding the Needles Without Sampling the Haystack

Every trace already has numbers attached to it - how long it took, how many tokens it used, how many times the agent called a tool, whether those tool calls failed. We watch those numbers over time.

If your agent normally finishes in 2 seconds and suddenly one takes 45 seconds, something went wrong.

If it usually makes 3 tool calls and one trace has 15, that's worth looking at.

We flag anything that lands way outside the normal range for that specific service and model, and flag them for review by our background AI agent.

Oodle's background AI agent compares these outlier traces and creates actionable reports with evidence traces for our customers to review and improve their agents.

Code Evaluators

Not everything needs an LLM to judge it. Some things you can just check.

Did the agent return valid JSON? Did it call the right tool? Did the response stay under a token limit? Did it hallucinate a URL that doesn't exist in your docs? These are yes/no questions - you can write a function for them.

We call these code evaluators. They run on every single trace, they're basically free, and they catch a surprising amount of bad behavior. If your domain has any rules that can be expressed as code - length limits, format checks, required fields, known-bad patterns - you should be evaluating those with code, not with an LLM.

We run code evaluators in a sandboxed environment for isolation and security.

Smart Sampling for Evals

Sometimes you genuinely need LLM-as-a-Judge. Checking for hallucination in a free-text answer, or whether a response was actually helpful - these need language understanding. There's no way around it.

But you don't need to run them on everything. And you definitely don't want to run them on a random sample.

Instead, we chain them. Sentiment flags a trace. Or anomaly detection flags it. Or a code evaluator flags it. Only then do we run the expensive LLM judge.

The cheap layers act as filters for the expensive layer.

You end up spending your LLM eval budget on maybe 5-10% of traces instead of picking a random 1%. And that 5-10% is the slice that actually has problems.

We're figuring this out too

This is early days for us, and honestly for everyone. If you're running agents in production and have figured out ways to catch silent failures without draining your eval budget, we'd genuinely love to hear from you.

Check out our live playground and let us know what you think! No signup needed: https://play.oodle.ai/genai/traces

See it in action on your own environment - signup for free at https://us1.oodle.ai/signup