Build representative and adversarial evaluation sets

Learning objective

Build an agent evaluation set from representative prompts, adversarial prompts, reference outcomes, expected tool behavior, and edge cases. Test both the final response and the actions that produce it.

Core idea

Treat each evaluation case as a small executable contract:

case = prompt + starting context + reference outcome + expected tool behavior
       + slice labels

The prompt can be one turn or a conversation. Starting context includes session state, tool fixtures, and retrieved content. The reference outcome states what must be true at completion. Expected tool behavior records required, allowed, and forbidden actions. Slice labels identify the task, risk, source, language, or failure mode.

An agent can return a polished answer after causing an unauthorized side effect. Two correct runs can use different wording or equally valid read-only paths.

Google’s Agent Development Kit (ADK) reflects this structure: an eval case can contain one or more turns with user content, expected tool use, intermediate responses, a reference final response, and initial session state. See the ADK agent evaluation guide.

How to build the set

1. Map tasks and risks before collecting prompts. Turn critical user journeys, tools, authorization boundaries, failure history, and prohibited outcomes into a coverage matrix. For example, cross search, modify, and cancel tasks with normal, ambiguous, unauthorized, and failure conditions.

2. Build the representative core. Sample privacy-reviewed production traces, then add requirements and domain-expert examples for important flows that are rare in logs. Vary wording, length, language, user role, history, and session state. Agent Platform recommends production-representative prompts, while Google’s responsible evaluation guidance emphasizes coverage and diversity. See Prepare your evaluation dataset and Evaluate model and system for safety.

3. Add adversarial cases from threat hypotheses. Try prompt injection through user input and untrusted retrieved content, attempts to override approval, unauthorized requests, secret extraction, malformed arguments, and inputs designed to cause excessive work. Include subtle paraphrases and obvious attacks. Adversarial means selected to expose a weakness; it does not mean merely uncommon. Google’s responsible evaluation guidance recommends explicit and implicit adversarial queries with varied length, formulation, tone, topic, and complexity.

4. Add edge cases around boundaries and dependencies. Cover missing or conflicting fields, empty retrieval results, zero and maximum values, duplicate requests, stale state, timeouts, permission denial, and partial success. An edge case can be benign; an adversarial case deliberately challenges the system.

5. Annotate the oracle. The evaluation oracle is the reference used to decide whether a run is acceptable. Record the required facts or state change, response constraints, and refusal or clarification behavior. Then record expected tool names, important arguments, required ordering constraints, forbidden calls, and side effects. Use an empty expected trajectory when correct behavior requires no tool. Agent Platform’s evaluation dataset fields include an optional reference and intermediate_events containing function calls and responses.

6. Review and preserve the set. Ask domain and security owners to validate high-risk cases. Remove duplicates, version cases with their agent contract and fixtures, and keep a held-out portion away from prompt tuning. Add observed failures without replacing the representative core.

Key distinctions and trade-offs

Reference response versus reference outcome: A reference response is one example of good wording. A reference outcome defines required facts, task state, and prohibited effects. Exact text is appropriate for structured or contractual output; semantic constraints are better when several phrasings are valid. A verified case with an approved reference is often called golden data. Golden describes annotation quality, not workload coverage.

Representative versus risk-weighted: A frequency-matched sample estimates typical behavior, but it can contain too few costly rare cases. Keep a production-like core and explicit risk slices. Report them separately instead of pretending an intentionally risk-heavy set represents traffic frequency.

Strict trajectory versus acceptable paths: Exact tool order can catch an unsafe route but can reject an equivalent implementation. Require order only when it changes correctness, authorization, or side effects. For example, check_approval must precede book_trip; independent catalog lookups need no fixed order. Include both outcome and behavior evidence where they matter.

Fixed versus simulated conversations: Fixed turns are reproducible. Simulation explores more paths but makes exact outcomes harder to prescribe. Use multi-turn cases for clarification, memory, and recovery.

Architecture / decision scenario

A corporate travel agent can search itineraries and book a trip only after the employee confirms a policy-compliant option. The team creates these slices:

The ordinary case protects the main journey. The ambiguous state and timeout cases protect operational boundaries. The injection and authorization cases protect trust boundaries. Fluent reference answers alone would miss an unauthorized booking or fabricated search result.

Common exam traps

• Calling a collection representative because it contains many prompts, without checking workload and slice coverage.
• Treating adversarial cases as unusual wording only, rather than attempts to cross trust, policy, privacy, or resource boundaries.
• Using one exact answer as the full oracle when several responses satisfy the task.
• Checking only the final response and ignoring tool selection, arguments, order, forbidden calls, and side effects.
• Requiring an exact trajectory when multiple safe paths are equivalent.
• Omitting “no tool call” expectations for clarification, refusal, and unauthorized requests.
• Interpreting a risk-heavy set as though it reflected production frequency.

What to remember

• Model each case as a prompt, starting context, reference outcome, expected tool behavior, and slice labels.
• Use representative cases for real workload coverage and adversarial cases to challenge known trust and policy boundaries.
• Treat benign boundary conditions as edge cases, even when no attacker is involved.
• Specify both required behavior and prohibited tool calls or side effects.
• Keep production-like and risk-weighted results visible as separate slices.

Knowledge check

References

• Google, Agent Development Kit: Agent evaluation
• Google Cloud, Prepare your evaluation dataset
• Google AI for Developers, Evaluate model and system for safety