Evaluate agent outcomes across execution levels

Learning objective

Evaluate agent task completion, tool usage, and multi-step reasoning with Amazon Bedrock Agent evaluations. AWS documents the agent-assessment service as Amazon Bedrock AgentCore Evaluations. Use that service’s trace and evaluator interfaces rather than assuming a model-response evaluation job measures a complete agent workflow.

Core idea

An agent can produce a polished response while failing to complete the requested work. It can also complete work through an incorrect or unsafe tool sequence. Evaluate the session outcome, individual responses, and tool calls separately, then combine their evidence according to the application’s acceptance policy.

AgentCore Evaluations supports agents hosted in AgentCore Runtime and agents hosted outside AgentCore. Its assessment depends on compatible telemetry, not merely on where the application runs. Multi-step reasoning assessment should focus on observable decisions, information dependencies, and outcomes. A trace or judge explanation does not prove the agent’s private reasoning is correct.

See 4.3.7 for reconstructing tool execution and 3.4.2 for explaining agent traces. This lesson uses execution evidence to judge whether the workflow succeeded.

How it works

Supply evidence at the right level

AgentCore reconstructs sessions from instrumented agent telemetry. AWS documents supported instrumentation using OpenTelemetry and OpenInference. For scoring, the service needs model prompts and completions plus tool inputs and outputs. Enable observability and CloudWatch Transaction Search as required by the telemetry setup; missing conversation content cannot be repaired by selecting another quality metric.

The levels have distinct meanings. A session covers the conversation across turns. A trace represents a request-response interaction. A tool span represents an individual tool call. Preserve those associations so a tool result remains connected to the request and context that justified it.

The Evaluate API takes evaluatorId and evaluationInput.sessionSpans. For targeted trace assessment, use evaluationTarget.traceIds; for tool-level assessment, use evaluationTarget.spanIds. Session-level evaluation does not need an explicit target because the service supports one session per on-demand evaluation input. Provide the surrounding session evidence even when targeting one tool call: the judge needs context to determine whether the action was appropriate.

Evaluate completion and tool choices

Use Builtin.GoalSuccessRate for session-level goal attainment. Its documented rubric assesses whether the assistant achieved all user goals using the conversation and tool evidence. For an explicit business contract, provide evaluationReferenceInputs with session-scoped assertions. These assertions can state that the agent confirmed required details, used a permitted tool, honored an approval dependency, and reported the actual outcome.

Assertions are judged statements, not enforcement controls. If a task changes external state, verify that state separately through the application’s test harness. A trace showing an attempted booking is insufficient evidence that a booking exists. Use the trace to connect the attempted action, tool response, and independently verified business result.

Use Builtin.ToolSelectionAccuracy for tool-level decisions. It examines whether the selected tool fits the request and preceding context. Use Builtin.ToolParameterAccuracy for parameter faithfulness. Its documented rubric focuses on whether parameter values come from preceding context and separates parameter faithfulness from tool selection and successful fulfillment. A correctly chosen tool can receive invented parameters; faithful parameters can be sent to an inappropriate tool.

For parameter requirements such as an exact account identifier, add an explicit harness check against the tool input in the spans. This deterministic check owns that application constraint; a judge’s assessment of plausibility should not replace identity or authorization validation.

Assess multi-step behavior against an explicit contract

Ground-truth evaluation adds evaluationReferenceInputs to Evaluate. A trace-scoped expectedResponse supports Builtin.Correctness, while session assertions support Builtin.GoalSuccessRate. Use these mechanisms to test whether later responses reflect earlier tool results and whether the agent satisfies cross-turn requirements. Builtin.Coherence assesses logical cohesion at trace level, including gaps and contradictions in the response. A clear explanation can still be factually wrong; coherence is not proof of correct tool execution.

For required tool sequences, supply expectedTrajectory.toolNames and choose among the documented session-level trajectory evaluators:

These trajectory evaluators use programmatic scoring rather than an LLM judge. They assess tool-name sequences, not the complete correctness of parameters or external effects. Choose exact order when every call is prescribed. Choose in-order matching when additional diagnostic calls are acceptable but required dependencies must hold. Any-order matching cannot prove that approval preceded a consequential action.

Keep the contract broad enough to admit valid alternative plans. If several paths safely reach the same goal, exact sequence matching can reject correct work. Conversely, if one step is mandatory for safety, an outcome-only metric can conceal its omission. Combine the sequence rule with parameter checks, completion evidence, and response correctness rather than selecting one score for the whole agent.

Check result completeness

Use on-demand Evaluate for targeted pre-release sessions and investigations. For sampled production traffic, CreateOnlineEvaluationConfig continuously monitors agent traces; rule.samplingConfig.samplingPercentage controls the percentage sampled, from 0.01 to 100. A value below 100 means evaluation is not exhaustive, so report the sampling policy and observed coverage rather than claiming every production interaction was assessed.

Evaluate returns evaluationResults, potentially containing multiple entries for traces or tool calls. AWS documents a maximum of 10 results per on-demand call, with the last 10 returned by default. Explicitly target bounded groups when you need evidence for more interactions. Do not assume a long session was fully assessed because the API call returned successfully.

Partial failures can coexist with successful results. Inspect failed entries for errorCode and errorMessage, and reconcile the returned session, trace, or span context against the expected targets. Track unscored targets independently from low-quality targets. See 5.1.7 for turning complete evaluation evidence into a release gate and 5.1.4 for handling judge disagreement.

Key distinctions and trade-offs

Goal attainment versus response quality: Builtin.GoalSuccessRate evaluates session goals. Builtin.Correctness evaluates a response at trace level. A good answer to one turn does not establish completion of all requested work.

Tool selection versus parameter faithfulness: Choosing the correct operation and supplying grounded arguments are different requirements. Neither guarantees that the downstream system accepted the operation.

Trajectory versus reasoning validity: Programmatic trajectory matching checks observed tool order. Judge-based assertions assess stated behavior. Neither reveals hidden reasoning or verifies an external state change by itself.

Strictness versus valid alternatives: Exact matching catches unexpected calls but rejects legitimate detours. In-order and any-order matching admit more behavior, so their acceptance policy needs complementary safety checks.

Architecture / decision scenario

A travel agent must confirm the destination, check availability, obtain approval, reserve a ticket, and report the reservation identifier. The test records the complete session and verifies the reservation in the test booking system.

Builtin.GoalSuccessRate receives assertions about completion and accurate reporting. Builtin.ToolSelectionAccuracy assesses selected tool spans. The harness checks the traveler identifier in tool parameters, and Builtin.TrajectoryInOrderMatch requires approval before reservation while allowing extra availability checks.

The candidate produces a confident confirmation but reserves before approval. Its final response can score well for correctness, yet the sequence gate fails. Another run obeys the order but returns no real reservation; completion fails. The team diagnoses each defect at its own evidence level instead of averaging both into a favorable overall agent score.

Common exam traps

• Treating a model-response evaluation as a complete agent assessment.
• Assuming AgentCore hosting alone supplies evaluable telemetry.
• Using a helpful final response as proof of business completion.
• Treating a faithful parameter as proof that the tool choice was correct.
• Using any-order matching to enforce a prerequisite.
• Ignoring the 10-result limit or partial failures in on-demand results.

What to remember

Use AgentCore Evaluations with complete, compatible session telemetry. Evaluate goals, responses, tool choices, and trajectories at their documented levels. Add deterministic checks for business state and exact parameters. Measure multi-step behavior through observable dependencies, then require complete assessment evidence before accepting an agent release.

Knowledge check

References

• AWS, How it works
• AWS, Telemetry setup and delivery
• AWS, Getting started with on-demand evaluation
• AWS, Prompt templates
• AWS, Ground truth evaluations
• AWS, Evaluate
• AWS, Create online evaluation
• AWS, SamplingConfig