AWSAWS Certified Generative AI Developer - Professional2026-100 Prerequisites

Read quality and runtime evidence separately

Completed by the knowledge check

Learning objective

Interpret evaluation metrics, logs, traces, and basic response quality signals. You should be able to identify what each signal measures, what it cannot prove, and which evidence you need before diagnosing a failed response. Detailed monitoring belongs to 4.3.1; evaluation dataset design belongs to 5.1.2.

Core idea

Operational evidence tells you whether the system executed and where resources or time were spent. Quality evidence tells you whether the resulting answer or action met the task’s acceptance criteria. A fast, successful model invocation can return an incorrect answer; a correct abstention can be a successful product outcome even when it contains no substantive answer.

Metrics summarize measurements, logs record events, and traces connect steps in an execution path. Evaluation scores apply a defined rubric to outputs. Interpret each signal with its unit, scope, denominator, and collection method before using it to make a decision.

How it works

Interpret operational metrics within their scope

For the bedrock-runtime endpoint, Amazon Bedrock publishes CloudWatch metrics including InvocationLatency, InputTokenCount, OutputTokenCount, and InvocationThrottles. InvocationLatency measures the time from request to receipt of the last token. It is not the full application journey when that journey also includes retrieval, tools, validation, and delivery to a user.

A latency percentile summarizes a distribution rather than one request. The median can remain acceptable while a smaller set of requests becomes very slow. Token totals can rise because there are more requests, longer inputs, longer outputs, or repeated attempts. Identify the relevant scope and traffic mix before calling that increase a quality regression. For streaming inference, output tokens per second (OTPS) separates slower token generation from longer responses: compute OutputTokenCount / (InvocationLatency - TimeToFirstToken) * 1000 with CloudWatch metric math. TimeToFirstToken is published only for ConverseStream and InvokeModelWithResponseStream, so this calculation applies only to those operations. See 4.3.3 for detailed interpretation.

Bedrock’s documented InvocationThrottles counting excludes throttled requests from Invocations, InvocationClientErrors, and InvocationServerErrors. Therefore, a ratio using only Invocations does not automatically describe all attempted requests. Retry settings can also affect observed throttle counts. The broader accounting rules belong to 4.3.3.

At the single-call boundary, Converse returns usage with token counts, metrics.latencyMs, and stopReason. Read completion signals before treating text as a finished answer. max_tokens indicates a token-bound stop, while tool_use indicates a tool request. end_turn indicates the model ended its turn; it does not certify factual correctness or business completion.

Use logs to inspect particular events

Logs help you inspect what happened for a particular request: its operation, configuration, inputs, outputs, or reported error. Bedrock model invocation logging can collect request data, response data, and metadata into CloudWatch Logs or Amazon S3. It is disabled by default, so absence of an invocation log does not establish absence of the invocation.

PutModelInvocationLoggingConfiguration configures the logging destination. The documented log format includes requestId, operation, and modelId. Those fields help you relate an observed result to the model call that produced it. Model invocation logging applies to supported calls through bedrock-runtime; it does not cover calls made through other endpoints such as bedrock-mantle. Keep that collection scope visible during an investigation.

Application logs supply decisions that a model invocation log cannot explain by itself: which source the application retained, why validation rejected an answer, and which product outcome it returned. Establish a correlation record between the application request and its downstream attempts. A retry can create several model invocations for one user-visible request; one log row is not necessarily one completed customer interaction.

Because invocation logs can contain model inputs and outputs, select diagnostic content deliberately and protect access. Privacy and sanitization are covered in 3.2.8 and 3.2.9.

Use traces to reconstruct the path

A trace connects steps so you can inspect ordering, dependencies, and intermediate observations. A log can say that a tool failed; a trace can help locate that failure after a model decision and before the next model call.

For Amazon Bedrock Agents, InvokeAgent accepts enableTrace: true. The response’s trace information maps to TracePart and includes agent and session identifiers alongside orchestration details. AWS documents trace information about action-group inputs, knowledge-base queries, returned outputs, and failure reasons. This is a specific Bedrock Agents mechanism, not an automatic trace for every custom agent framework.

Trace content describes observed execution and can include model-generated rationale. Do not treat that rationale as independently verified proof that an action was correct. Confirm a consequential action using its tool result and business completion evidence. Public explanations and agent execution evidence are covered in 3.4.1 and 3.4.2.

Read response-quality signals as separate dimensions

Bedrock RAG evaluation provides retrieve-only and retrieve-and-generate metrics. For retrieval, Builtin.ContextRelevance measures how relevant retrieved text is to the question. Builtin.ContextCoverage measures coverage of the ground truth information and requires ground truth in the prompt dataset. Relevant passages can still omit a necessary fact.

For generated responses, Builtin.Correctness measures answer accuracy, Builtin.Completeness measures how fully the answer resolves the question, and Builtin.Faithfulness measures avoidance of hallucination relative to retrieved text. These are evaluator-model judgments under defined metrics, not interchangeable probabilities of truth. A faithful answer can reproduce an incorrect source, and a correct sentence can leave half the question unanswered.

Builtin.CitationPrecision assesses how many cited passages are correctly cited, while Builtin.CitationCoverage assesses support from citations and missing citations. Separate source attribution from fluency: polished text is easier to read but does not establish support. Schema validity, successful parsing, and a model’s own confident wording are also insufficient quality signals on their own.

Key distinctions and trade-offs

Choose evidence by the question you need to answer:

Question	Evidence to inspect	Limit
Did inference slow down?	Latency distribution and individual call timing	Does not identify answer correctness
What input produced this answer?	Correlated invocation and application logs	Only covers configured collection scope
Which step caused the failure?	Execution trace and tool observations	A recorded rationale is not proof of correctness
Did retrieval include necessary evidence?	Relevance and coverage evaluation	Does not prove generation used it correctly
Is the answer acceptable?	Reviewed quality rubric and completion checks	Depends on dataset and evaluation method

Keep denominators visible. “Ninety percent passed” is incomplete without knowing whether the sample includes errors, refusals, unanswerable requests, and repeated attempts. Results also need applicable slices: a strong aggregate can hide weak performance on consequential questions. Detailed metric publication belongs to 4.3.4.

Architecture / decision scenario

A support assistant becomes slower after a prompt change, while users report unsupported answers. The team treats these as two questions requiring connected but distinct evidence.

CloudWatch InvocationLatency and per-request output-token measurements show increased latency and longer responses. In a controlled streaming reproduction, the team checks OTPS against its baseline to distinguish an output-length change from slower token generation. Correlated logs reveal that the changed prompt requested more detail. Bedrock Agents traces, enabled through InvokeAgent, show which knowledge-base queries and action-group calls occurred before the answer. The team checks those observations rather than accepting the agent’s explanation of its own success.

Evaluation then finds high Builtin.ContextRelevance but weak Builtin.ContextCoverage: retrieval found related material while missing the policy exception. Answer review also finds unsupported claims, reflected in faithfulness evaluation. Reducing response length can address latency, but it cannot replace missing evidence. The team repairs the retrieval gap and repeats quality evaluation while monitoring runtime timing separately.

Common exam traps

Treating a successful HTTP response or end_turn as answer acceptance.
Equating invocation latency with full application latency.
Inferring no invocation occurred because optional logging was disabled.
Treating every model attempt as a separate completed user request.
Confusing relevance, coverage, correctness, and faithfulness.
Treating fluent text or a model-generated rationale as independent verification.

What to remember

Runtime metrics measure execution; quality metrics assess task outcomes.
Logs describe events; traces connect the execution path.
Inspect collection scope and counting rules before interpreting aggregates.
Check completion, evidence support, and question coverage separately.
Use correlated telemetry to locate defects, then evaluation to verify the repair.