Building an AI agent almost always starts with a moment of genuine excitement. You write a system prompt, give the model two or three tools, and run a few test scenarios in a chat window. It works. The agent reasons through the user request, calls the right API, and returns a clean, structured answer. It feels like magic.
Then the system meets reality. The moment you put that workflow into production, the illusion cracks. The model misinterprets an ambiguous user query, calls a mutation tool with hallucinated parameters, or gets stuck in a retry loop until downstream timeouts fire. You watch an agent fail live in front of a customer, and a familiar sinking feeling sets in.
When traditional software breaks, our engineering reflex is immediate: reproduce the bug, write a failing unit test, patch the code, and verify that the test turns green. With agentic systems, however, engineering teams routinely freeze. Instead of writing automated tests, developers fall back on manual spot-checks in a web playground, testing two or three friendly prompts, hoping for the best, and crossing their fingers before pushing to production.
Inverting the Testing Model
This hesitation does not come from a lack of engineering discipline. It comes from an uncomfortable reality: our traditional testing muscle is built entirely on the assumption of strict determinism. Given an explicit input X, a function must produce an exact output Y. For instance: if calculateTotal(items) returns $104.50 on one run and $102.00 on the next, the code is broken. But when an engine generates natural language, expecting exact string equality makes test suites brittle and useless.
Years ago, when I introduced property-based testing at Block, I watched teams navigate that exact same hesitation. Traditional tests rely on hand-picked, comfortable inputs. Property-based testing turned that model upside down: instead of static values, it generated hundreds of randomized inputs to verify that core business invariants never broke.
At first, adoption was an uphill battle. But the pushback was not about code quality; it was about developer psychology. Engineers were far more anxious about test flakiness and noise in CI than about users encountering edge-case bugs in production. It is a fundamental truth of software engineering: developers will tolerate unverified code in production far longer than they will tolerate a test suite that feels unpredictable in their daily workflow.
Autonomous agents demand that exact same mental leap, but inverted. With property testing, we managed deterministic code against randomized inputs. With AI agents, we manage deterministic logic around probabilistic outputs.
This distinction is the central thesis of agentic engineering: autonomous agents should be tested as probabilistic programs operating within deterministic boundaries. You do not attempt to control the model's exact wording; you establish the invariant boundaries it must respect, and evaluate how effectively it achieves its behavioral objective within those bounds.
When developers first confront this non-determinism, their initial defense is often to mock the language model in unit tests. We write a mock that returns the exact JSON payload we expect, run our assertions, and watch the build turn green. But mocking an LLM gives a false sense of security: it tests your hardcoded assumptions rather than the agent's actual reasoning, tool selection, and error recovery. In production, the model does not follow your mock.
To bridge this gap, we have to stop testing for exact words and start evaluating behavior, operational constraints, and execution paths. To help teams build this muscle without reinventing the wheel, I open-sourced evalkit, a lightweight toolkit designed to make writing and running agentic evaluations straightforward.
Autonomous agents should be tested as probabilistic programs operating within deterministic boundaries.
Why shouldn't we just mock the LLM in our integration tests like we do for databases and third-party APIs?
Mocking the LLM only tests your mock.
In traditional software, databases and payment gateways follow strict, deterministic contracts. In an agentic system, the model is not an external dependency with a fixed contract; it is the reasoning engine responsible for tool selection, prompt interpretation, and error recovery. Mocking the model gives you a green test suite that verifies nothing about how your agent behaves in the wild.
Measuring Probabilistic Systems
To navigate this paradigm shift, it helps to draw a clean architectural distinction between a test and an eval:
- A test asks: Did this invariant hold? It is binary, deterministic, runs in microseconds, and tolerates zero failure. If an agent calls an unauthorized mutation tool, leaks credentials, or exceeds its step limit, the test fails immediately.
- An eval asks: How effectively did the system achieve its behavioral objective? It is qualitative, scored across repeated trials, and accounts for natural variation in phrasing, reasoning, and tone.
A mature agent architecture does not replace tests with evals; it nests tests inside evals. Deterministic tests enforce the non-negotiable boundaries, while probabilistic evals measure performance and quality within those boundaries.
Manual playground checks create a dangerous illusion of stability. An engineer runs two or three cooperative queries, receives good responses, and assumes the prompt is ready for deployment. But a language model tested against clean inputs in isolation behaves very differently when subjected to ambiguous user intent, malformed external payloads, and multi-step tool failures. Without automated baselines, prompt engineering becomes guesswork: tweaking a sentence to fix one user scenario quietly breaks three others. As a result, engineering teams spend their cycles firefighting regressions in production rather than catching them in pull requests.
Automated evals provide the quantitative guardrails required to engineer agentic software with confidence:
- Establishing measurable baselines: quantifying an agent's success rate, step count, and latency before shipping changes to users.
- Preventing silent regressions: guaranteeing that optimizing a prompt for one domain does not degrade performance across existing workflows.
- Benchmarking model transitions: providing clear empirical proof when evaluating a migration from one model family to another (such as comparing Claude 3.5 Sonnet to Gemini 2.0 Flash or GPT-4o).
- Detecting systemic failure modes: identifying recurring issues like tool-calling loops, invalid JSON payloads, hallucinations, and prompt injection vulnerabilities.
The long-standing engineering principle of "Think Big, Ship Small" applies directly here. We do not want to boil the ocean by attempting to evaluate every theoretical nuance of human language on day one. Instead, the strategy is to establish a small, representative evaluation suite early, chipping away at coverage as real edge cases surface in production.
Can setting temperature: 0 make our agent evaluations completely deterministic?
temperature: 0 make our agent evaluations completely deterministic?
No, and more importantly: exact string determinism is the wrong goal for AI testing.
While setting temperature to zero minimizes randomness, cloud AI providers often show slight variations across runs. But even if a provider offered 100% byte-for-byte reproducibility under zero temperature, testing for exact string equality remains fundamentally flawed.
In production, real users write varying prompts, and an agent that responds "I have cancelled appointment #9842" on one run and "Appointment #9842 has been removed" on the next is functioning correctly. If you build test suites around rigid string equality, your builds will either fail on harmless phrasing changes or pass without verifying substance.
Testing agentic software requires evaluating behavioral goal completion and invariant boundaries, not asserting exact words.
How large should our golden dataset be before we can trust our evaluation suite?
Start small: 20 to 50 curated tasks is plenty for an initial baseline.
Many teams stall out trying to handcraft hundreds of synthetic edge cases before shipping their first prompt. A compact suite covering core happy paths, primary failure modes, and critical safety constraints provides the vast majority of your signal. From there, expand organically: whenever production monitoring flags an unhandled failure or edge case, sanitize the request to remove sensitive data and credentials, turn it into a test case, and add it to your golden dataset as a permanent regression test.
Where Evals Run: Offline vs. Online
When structuring an evaluation strategy, testing divides naturally into two operating environments: offline evaluations (before code merges) and online evaluations (after code reaches production). Both environments rely on the same core asset: the execution trace. Think of an execution trace as the agent's complete flight recorder, capturing the user's initial prompt, the agent's internal thoughts, every tool it called, the raw data returned, and the final response.
| Evaluation Class | Environment | Analogy | What It Validates | Trade-offs & Scope |
|---|---|---|---|---|
| Isolated Evaluations | Offline (Local / CI) | Unit Tests | Confirms prompts and skills trigger expected outcomes. Enables mocking tool responses to test edge flows that are difficult to reproduce end-to-end. | Fast, inexpensive, and highly targeted. Does not verify multi-step orchestration or UI state rendering. |
| Harness Evaluations | Offline (CI / Staging) | Integration Tests | API-level tests verifying that the overall agentic system operates as intended across its multi-step execution loop. | Tests end-to-end tool chaining and state management. Slower and more token-intensive than isolated evaluations. |
| UI Evaluations | Offline (Staging / E2E) | E2E Tests | Confirms the full behavior of the agentic task through the UI, guaranteeing custom widgets, interactive cards, and streaming states mount cleanly. | High end-to-end fidelity. Slower, more brittle to frontend visual updates, and harder to parallelize. |
| Online Evaluations | Online (Production) | Production Monitoring & Health Checks | Confirms through automated scorers and monitors that real users are having a reliable, high-quality experience. | Real-world validation against live user traffic. Runs asynchronously on a small sample (such as all errors plus 2% to 5% of normal traffic) to keep API costs and latency under control. |
The relationship between offline and online testing forms a continuous feedback loop. Offline evaluations run in local environments and CI pipelines, progressing outward from fast isolated prompts to multi-step orchestration harnesses and browser sessions. They catch syntax errors, broken tool schemas, and prompt regressions before a pull request merges.
Online evaluations take over once code ships. Because evaluating live requests in the critical path would add unacceptable latency, production monitoring runs asynchronously through background queues. High-volume architectures evaluate 100% of errors and user cancellations alongside a 2% to 5% sample of nominal traffic. Whenever an unhandled failure surfaces in production, the sanitized trace is turned directly into a permanent test fixture in your offline suite.
What is the difference between an isolated eval and a harness eval?
Think of isolated evals as unit tests and harness evals as integration tests.
Isolated evaluations test individual components (a single prompt, a skill definition, or a tool argument parser) with mocked tool responses, making them fast and cheap to run locally. Harness evaluations execute the full live agent loop across multiple turns, requiring the model to invoke live tools, handle intermediate errors, and manage memory state. Run isolated evals on every commit; run harness evals in CI before staging and production deployments.
The Anatomy of an Eval Suite and Harness
To an engineer who has never written an eval, the terminology can feel abstract until you see an actual suite file. In evalkit, evaluations are declared in high-level YAML files that live directly in your repository alongside your code.
Here is what a representative task looks like for an appointment management agent:
suite: appointment-management
version: 1
defaults:
max_steps: 6
max_latency_ms: 15000
tasks:
- id: cancel-confirmed-appointment
vars:
appointment_id: "apt_9842"
client_name: "Sarah Jenkins"
prompt: "Cancel appointment ${appointment_id} for client ${client_name}."
tags: [appointments, cancellations]
severity: high
expect:
# Deterministic process checks (fast, free, zero variance)
tools:
required:
- name: get_appointment
arguments:
appointment_id: "${appointment_id}"
- name: cancel_appointment
arguments:
appointment_id: "${appointment_id}"
forbidden:
- issue_refund
order:
- get_appointment
- cancel_appointment
max_calls:
cancel_appointment: 1
max_steps: 4
# Deterministic output checks (regex and substring matching)
must_contain:
- "${appointment_id}"
- "${client_name}"
must_not_contain:
- "Internal Server Error"
- "Traceback"
patterns:
- "(?i)cancel(l)?ed"
# Qualitative semantic check (LLM-as-a-Judge)
rubric: |
- Communicates the cancellation with empathy, maintaining a polite and professional tone.
- Proactively offers next steps or assistance with rescheduling without being pushy.
- Avoids leaking internal system terminology, database IDs, or backend error codes to the client.Breaking down this structure reveals how closely it maps to deterministic testing conventions:
- Suite and contract version (
suite,version): The top-level container and its version. If you alter grading rubrics or prompts, bumping the version prevents you from comparing apples to oranges across historical runs. - Task identity (
id): The unique identifier across all test runs. Think of it as your test method name (test_cancel_appointment). Without a stable ID, you cannot track whether a specific workflow regressed over time. - Dynamic variables (
vars): Instead of hardcoding a single static ID and tailoring your prompt to only one scenario,evalkitsupports dynamic variables. On each run, the agent encounters realistic variations in names and IDs, guaranteeing the test reflects real-world conditions while remaining repeatable. - Deterministic process checks (
expect.tools,expect.max_steps): These verify that the agent takes the expected actions:tools.required: Confirms the agent invoked necessary backend capabilities and passed expected structured arguments (verifying thatcancel_appointmentreceived the resolved${appointment_id}).tools.forbidden: Establishes critical safety guardrails. If an agent calls a forbidden tool (such as issuing an unauthorized refund), the entire task immediately fails with a score of 0, regardless of how polite or complete the generated text appears. In production, this evaluation check complements runtime security controls like API scopes and user permissions.tools.order: Verifies step order (the agent must fetch the appointment before attempting to cancel it).tools.max_calls: Guards against infinite loops and retry storms.max_steps: Enforces an operational budget so the agent does not spin through six intermediate thoughts for a simple action.
- Deterministic output checks (
must_contain,must_not_contain,patterns): Fast code assertions that run instantly. They verify that the resolved ID and client name appeared in the answer, check cancellation confirmation via regex, and guarantee that no backend stack traces leaked to the user. - Model-based checks (
expect.rubric): An LLM judge uses a fast model to evaluate subjective qualities that traditional code cannot check, such as tone, empathy, and conciseness. To keep grading reliable, give the judge clear yes-or-no criteria rather than open-ended 1-to-5 ratings. Breaking expectations into individual pass/fail checks turns subjective grading into stable, reproducible metrics supported by quotes from the transcript.
In practice, the vast majority of evaluation checks in a task-oriented agent can, and should, be regular deterministic code. You do not need an LLM to verify tool calls, count steps, validate JSON schemas, or check substring bounds. Programmatic assertions do the heavy lifting for free, reserving expensive judge models strictly for qualitative semantic grading.
The vast majority of evaluation checks in a task-oriented agent can, and should, be regular deterministic code.
Once you have established deterministic process assertions and bounded semantic rubrics, the next challenge is operational: how do you interpret scores when the underlying system is inherently probabilistic?
Should I use an LLM-as-a-judge to verify if my agent called the correct tool?
No. Tool call verification is completely deterministic.
While an agent's reasoning is probabilistic, its tool calls produce structured, predictable data. Checking whether the agent called cancel_appointment, validating arguments against a JSON schema, enforcing step limits, or verifying execution order should always be handled in code. Deterministic assertions run in microseconds, cost nothing, and never hallucinate. Reserving LLM judges strictly for qualitative attributes (tone, conciseness, and politeness) keeps your evaluation pipeline fast, inexpensive, and reliable.
Do we really need to measure infrastructure metrics like latency and step count in our evals?
Yes, measuring infrastructure metrics is just as critical as measuring response accuracy.
When you upgrade models or tweak agent prompts, you can easily introduce extended reasoning loops, verbose retries, or redundant tool calls. Even if the agent eventually produces the correct answer, latency skyrockets and downstream timeouts multiply. All of this hits end users directly in production. By setting explicit thresholds on latency, step counts, and token budgets in your evaluation harness, you catch regressions and receive clear warnings early, rather than failing silently in the wild.
Taming Randomness: The Three Rules of AI Testing
In a deterministic test suite, history is largely irrelevant: a commit is either green or red. If a unit test passes on main, you rarely need to examine what happened three weeks ago.
With AI models, a single test run does not tell the whole story. A prompt that scores 92% today might score 89% tomorrow without a single line of code changing, simply due to natural variation in responses. Because of this, tracking results over time is essential for operating AI systems with confidence.
Rule 1: The 3-Run Smoke Test
When testing an agent, never rely on a single run to declare success. In my workflows, I make it a rule to run every prompt or evaluation at least 3 times.
Here is why a single run is deceptive:
- Catching flakiness immediately: Running a test 3 times will not prove that a prompt is 100% bug-free. In fact, an unreliable prompt that fails 30% of the time will still pass 3 consecutive runs about a third of the time. But 3 runs work brilliantly as an asymmetric smoke test: if even one run fails (yielding
[PASS, FAIL, PASS]), you immediately know the agent is unstable before merging your pull request. It will not prove perfection, but a single failure instantly unmasks instability. - Fast feedback without burning budget: Running an evaluation 20 times gives you high statistical certainty, but it multiplies API costs by twenty and slows down CI. Three runs strike an effective balance during active development: fast enough to run locally in seconds, but thorough enough to catch brittle prompts before you push code.
Rule 2: Measure Your Noise Band
To understand your evaluation results, you also need to know how much your score fluctuates naturally. Run your evaluation suite twice on the exact same code commit.
If your suite has 10 tasks, a single task flipping from pass to fail swings your overall score by 10% (dropping from 90% down to 80%). If your suite has 50 tasks, that single swing is 2% (from 90% to 88%).
When consecutive runs on an unchanged codebase fluctuate within this range, you are looking at your baseline noise band. Any score changes inside this band are normal probability swings, not a true regression or a breakthrough. Knowing this baseline prevents teams from wasting hours chasing phantom bugs or celebrating lucky coin flips.
Rule 3: Compare Changes Head-to-Head
When you optimize a prompt or upgrade to a newer model across 30 or more tasks, looking only at aggregate average scores can mislead you. Some tasks are straightforward while others are subtle edge cases. If an easy task happens to pass while a critical workflow breaks, your overall average might stay identical even though the agent regressed.
To get a clean signal, evalkit compares versions task by task on the exact same inputs:
Task ID Baseline (v1.2) Candidate (v1.3) Delta Step Delta
-------------------------------------------------------------------------------------------------
apt-cancel-confirmed [PASS, PASS, PASS] [PASS, PASS, PASS] 0 -0.2 steps
apt-reschedule-conflict [PASS, FAIL, PASS] [PASS, PASS, PASS] +33% (Fix) -1.1 steps
apt-refund-unauthorized [PASS, PASS, PASS] [PASS, PASS, PASS] 0 0.0 steps
apt-multi-slot-search [FAIL, FAIL, FAIL] [PASS, PASS, FAIL] +67% (Gain) +0.3 steps
apt-edge-timezone-rollover [PASS, PASS, PASS] [FAIL, PASS, FAIL] -67% (REG) +1.8 steps
-------------------------------------------------------------------------------------------------
Aggregate Suite Pass Rate: 73.3% 80.0% +6.7% (Net)Looking at the table above demonstrates why head-to-head diffs are essential: even though the candidate prompt increased the aggregate pass rate from 73.3% to 80.0%, the paired comparison immediately unmasks a critical regression on apt-edge-timezone-rollover.
- Evaluating individual deltas: Compare how each specific task performed before and after. If both versions pass or both fail, nothing changed. This isolates only the tasks where the updated prompt altered behavior.
- Auditing operational regressions: For instance: a prompt tweak might keep your success rate intact while quietly doubling the number of tool calls needed to complete a task. Tracking step counts and speed guarantees you do not ship changes that make your system sluggish or expensive.
- Confirming genuine capability gains: Verify that task-level wins outnumber losses across repeated runs, confirming that an improvement is statistically authentic rather than a random probability swing.
That said, tracking historical metrics is only valuable if teams focus on real outcomes rather than vanity scoreboards. Artificially inflating evaluation pass rates by simplifying test prompts defeats the purpose of the harness. The goal is to build genuine confidence in system reliability.
If I change the agentic rubric, should I create a new baseline forward or rescore past runs?
It depends on whether you preserved your execution traces.
If you save the raw execution traces (every model turn, tool call, and tool output), you do not need to re-run the full multi-step agent workflow. You can simply run your updated LLM judges against the recorded traces under a new version (version: 2). If you forgot an important criterion in your rubric or want to tighten tone guidelines, you can re-score historical runs without burning time and tokens re-executing tool calls. This is why saving full execution traces is so valuable.
How do we handle flaky evaluations in CI without setting arbitrary pass-rate thresholds?
Enforce hard gates on critical workflows rather than loose aggregate averages.
Relying on a single average score can hide regressions behind unrelated wins: an agent could completely break its primary workflow while passing five minor edge cases, keeping the average score steady. Instead, use hard gates for critical paths: core business actions and safety rules (tools.forbidden) must pass 100% of the time across all runs. For subjective text quality, compare results directly against your baseline to confirm genuine improvement rather than relying on loose pass rate thresholds.
Building an Eval-Driven Architecture
Bridging the gap from deterministic testing to AI evals does not mean throwing away existing engineering instincts or abandoning rigor. It means elevating those instincts to a higher level of abstraction: applying systematic verification to execution paths, performance budgets, and output quality across multiple runs rather than expecting single identical return values.
To prevent evaluations from becoming a developer bottleneck, successful engineering organizations structure execution across four distinct operational tiers:
- Tier 1 (Fast pre-merge smoke, under 2 minutes): Run isolated evaluations with mocked tools across a curated smoke suite of 5 to 10 core tasks using the 3-run rule. Because it completes in under two minutes without spending live API tokens, developers get instant feedback on syntax and broken schemas without ever feeling that AI testing is slowing down their pull requests.
- Tier 2 (Comprehensive staging harness, 10 to 20 minutes): Execute your full 50+ task golden dataset against sandboxed tools, running 5 to 10 iterations per task. This tier runs head-to-head comparisons across runs, detects subtle prompt regressions, and audits step counts without slowing down daily pull request reviews.
- Tier 3 (Continuous online monitoring): Stream production traces asynchronously into decoupled background queues, evaluating 100% of errors and a 2% to 5% sample of nominal traffic to detect model drift and discover new edge cases.
- Tier 4 (Automated regression curation): When production monitors detect novel failure modes, sanitize the traces to remove sensitive data and credentials, turn them into test fixtures, and add them directly to Tier 1 or Tier 2 suites.
Write Once, Run Everywhere
This tiered structure unlocks an essential architectural multiplier: evaluator reuse across the lifecycle. A deterministic schema check or an atomic rubric judge written for pre-merge CI runs plugs directly into your production queue to evaluate sampled live traces. By keeping evaluation harnesses DRY, your definition of quality stays consistent from local development through production monitoring.
Without an evaluation suite, every prompt tweak or model migration feels like walking through a minefield. Once you have an automated harness in place, the dynamic changes entirely. Your team moves with confidence, knowing that their work is verified against an empirical baseline rather than hopeful assumptions.
As our systems become more agentic, our role as engineers shifts from writing rigid logic to establishing boundaries, designing test harnesses, and verifying reliability. By treating evaluations with the same rigor we have traditionally brought to deterministic testing, we guarantee that our systems remain reliable, maintainable, and trusted by the people who rely on them every day.
By treating evaluations with the same rigor we bring to deterministic testing, we guarantee that our systems remain reliable, maintainable, and trusted.
To help teams navigate the practical edge cases of implementing this architecture, I have compiled answers to the most frequent questions engineers and engineering leaders ask when establishing their evaluation pipelines.
Can I reuse the same evaluators across offline test suites and online production monitoring?
Yes, and keeping your evaluators DRY is one of the biggest architectural advantages.
A deterministic schema check or an LLM rubric judge created for pre-merge CI runs can be plugged straight into your production queue to evaluate sampled live traces. When you write validation logic once and share it between local harnesses and production monitors, your definition of quality stays consistent everywhere.
How do we prevent running evaluations from becoming a massive bottleneck in our CI pipeline?
Adopt a tiered pipeline and parallelize task execution across worker pools.
Running 50 multi-step agent harness evaluations with three iterations each means executing 150 multi-step agent runs per pull request. If run sequentially, CI times will quickly balloon to 30 minutes or more. You prevent this by combining three practical techniques: gating pre-merge pull requests strictly with fast smoke suites (5 to 10 critical tasks with mocked tools) that finish in under two minutes; parallelizing task execution up to your model provider's concurrency limits; and running fast programmatic checks (tool order, argument schemas, step limits) before triggering LLM judges. If a deterministic check fails, the harness fails the task immediately and skips the judge model entirely. Full golden suites run post-merge or in nightly staging pipelines.