Key findings

• Test the complete workflow, not only the language model response.

• Keep a holdout set and add real failures to a separate regression suite.

• Use the simplest reliable grader for each criterion.

• Measure important slices, serious errors, review effort, latency, and cost.

• Re-run gates after changes to models, prompts, data, tools, policy, or vendors.

Define the evaluation contract

Write the task, intended users, allowed and disallowed behavior, inputs, expected output structure, tool actions, and fallback. Establish the current process baseline. Then set launch criteria for task quality, serious errors, policy compliance, latency, reliability, variable cost, and human review. Without a precommitted contract, teams tend to redefine success around whatever the latest demo does well.

Separate model quality from workflow outcome. A correct extraction can still be written to the wrong customer; a good draft can disclose restricted data; a useful answer can arrive too late; and an accurate classification can create harm if the routing policy is wrong. NIST TEVV guidance emphasizes context because the appropriate measurements depend on how a component is used.

Build a representative, versioned test set

Sample real workflow cases lawfully and protect personal or confidential data. Cover ordinary frequency, business value, difficult edge cases, missing and contradictory inputs, multiple languages or formats, and cases with higher consequence. Add synthetic cases only to fill known gaps; do not let easy generated examples replace real operational diversity.

Create a separate adversarial set for prompt injection, data exfiltration, unauthorized actions, harmful requests, malformed files, oversized input, poisoned retrieval content, and tool manipulation. Keep a holdout set that prompt and workflow authors do not repeatedly optimize against. Maintain provenance, expected behavior, severity, slice labels, and version history. Add production failures to a regression set after review.

Use deterministic tests wherever possible

Apply exact assertions to JSON schema, required fields, enum values, calculations, date and amount bounds, recipient allowlists, data filters, role checks, idempotency keys, retry limits, tool arguments, and downstream writes. Unit-test each deterministic component independently. Mock provider and tool responses to exercise timeouts, malformed output, rate limits, partial failure, and duplicate delivery.

Run integration tests through the entire pipeline in a sandbox with production-like authentication and permissions. Verify that an unauthorized model proposal is denied, rejection prevents execution, and a retry cannot duplicate a payment or message. Google production-ML guidance recommends testing data, model quality, serving infrastructure, and integration, and validating new versions against both the previous version and fixed thresholds.

Evaluate subjective output with explicit rubrics

Define criteria separately: factual correctness, groundedness, completeness, relevance, instruction adherence, tone, safety, and policy compliance. Use clear ordinal labels with examples of pass, minor defect, major defect, and critical failure. Ask domain experts to label a calibration sample and measure agreement. Resolve ambiguous policy before scaling annotation.

Use code-based graders for objective facts, reference matching where a canonical answer exists, human graders for high-consequence or nuanced judgment, and model graders for scale only after validating them against human labels. A model grader can share blind spots with the system under test. Record grader model, prompt, version, and uncertainty, and periodically recalibrate with blinded human review.

Test trajectories, security, and operational failure

For an agent, assess both final answer and trajectory: tools selected, order, arguments, data accessed, policy decisions, retries, and stopping condition. A correct final response does not excuse unauthorized access or an unsafe intermediate action. Enforce maximum steps, tokens, duration, spend, and tool calls. Test compromised documents and tool responses because indirect prompt injection arrives through trusted-looking workflow content.

Exercise provider outage, slow response, connector change, stale retrieval, unavailable reviewer, queue backlog, lost event, and partial downstream commit. Confirm fail-safe behavior, fallback, alerting, rollback, and manual continuation. Test data deletion and identity revocation as operational capabilities. OWASP recommends structured adversarial testing before production and after material changes to prompts, tools, memory, retrieval, policy, or providers.

Analyze slices and uncertainty

Report results by important case type rather than one average. Relevant slices might include document source, language, customer segment, product, request intent, input length, reviewer team, or risk tier, provided segmentation is lawful and meaningful. Set minimum sample expectations and show confidence intervals or explicit sample limitations where statistical claims are made.

Track critical failures as counts with case review. A system with a high average score can still be unfit if it occasionally sends restricted information or performs an unauthorized action. Compare candidate and current versions on the same frozen set. Investigate regressions even when the aggregate improves. Keep the old baseline visible so incremental prompt tuning does not hide deterioration over several releases.

Create a release gate

The release record should include code, model, prompt, retrieval, tool, policy, data, and grader versions; test results; unresolved defects; risk acceptance; rollback target; and approvers. Block release when a critical scenario fails, mandatory controls are absent, quality drops below threshold, cost exceeds the operating envelope, or fallback cannot be demonstrated.

Use staged rollout after offline tests: shadow, internal users, limited traffic, then wider production. Compare each stage to acceptance criteria. Do not allow a vendor model update or prompt edit to bypass the gate. Small changes can interact with retrieval and tools in unexpected ways, so define what is material and automate regression runs for every deployment where feasible.

Continue evaluation in production

Offline data becomes stale and cannot represent every real condition. Sample production runs for human review, collect user corrections, join delayed outcomes, monitor policy events and quality proxies, and watch shifts in input mix. Protect sampled content through minimization, access control, and retention. Use canary releases to compare versions on a small fraction of traffic before full replacement.

Set triggers for rollback or reassessment: serious incident, quality or slice regression, rising review rate, unexpected tool behavior, latency or cost increase, new data source, or legal change. Evaluation is a feedback system. Production failures improve the test suite; the test suite protects the next release; and monitoring verifies whether laboratory performance survives real operation.

Sources and methodology

This article synthesizes the primary sources below as of the publication date. Forecasts and recommendations are directional scenarios, not guarantees; they should be tested against your workflow, data, risk tolerance, and current vendor documentation.

National Institute of Standards and Technology: AI test, evaluation, validation and verification (accessed 2026-07-30)

National Institute of Standards and Technology: Generative Artificial Intelligence Profile (accessed 2026-07-30)

Google for Developers: Production ML systems: deployment testing (accessed 2026-07-30)

Google AI for Developers: Evaluate model and system for safety (accessed 2026-07-30)

OpenAI: OpenAI Evals API Reference (accessed 2026-07-30)

OWASP Foundation: OWASP AI Agent Security Cheat Sheet (accessed 2026-07-30)