AWSAWS Certified Generative AI Developer - Professional2026-104.3 Implement monitoring systems for GenAI applications

Preserve request histories that explain changes

Completed by the knowledge check

Learning objective

Track forensic request histories, user actions, and changes in model behavior for operational and compliance investigations.

Core idea

A forensic history must answer what happened to a particular request, who initiated it, which configuration executed it, and what evidence supports the conclusion. A latency graph cannot reconstruct an individual answer. A stored answer cannot establish who authorized the action that followed it.

Build a history from distinct evidence layers: application events for user intent and business decisions, CloudTrail for AWS API activity, and Bedrock invocation logs for supported model inputs and outputs. Preserve the configuration and source revisions that connect those events. See 3.3.4 for the governance evidence model and 4.3.2 for invocation-log configuration; this lesson focuses on tracking a request across time and explaining changed behavior.

Completeness is a design property. If you capture only failed requests, you cannot reliably compare a failure with successful requests that used the same prompt. If you capture only model calls, you miss an authorization denial before inference or a business action after generation.

How it works

Join histories through retained evidence

Use the request, attempt, principal, and retained-evidence contract in 3.3.4 to join application events, CloudTrail records, and protected object versions.

For a supported model call, pass a non-sensitive attempt identifier through Converse requestMetadata or the X-Amzn-Bedrock-Request-Metadata header for InvokeModel. It appears in invocation logs only when logging is enabled in the call’s Region. When signing the header yourself, include it in the Signature Version 4 SignedHeaders; AWS SDKs that expose metadata as a parameter handle signing automatically. Keep the service requestId separate from the identifier your application supplies.

Record submitted, authorization denied, answer rejected, answer delivered, and business operation confirmed as distinct application events. Those events explain user actions that AWS API records cannot reconstruct by themselves. A shared application role identifies an AWS caller, not the person who selected an option. Preserve the authenticated user association from the application boundary.

Distinguish an event that requests an action from one that confirms its effect. For an investigation into a recommendation, identify what the person received and selected, then whether the backend accepted and completed the requested operation. A missing confirmation is an unresolved outcome, not evidence that nothing happened.

Build a configuration-change timeline

CloudTrail records the remaining Bedrock API operations, including configuration changes, as management events. Use that evidence to build a change timeline alongside the runtime request history. The timeline answers who changed a resource and when; each request’s release identity answers which configuration actually executed.

UpdateAgentAlias is a concrete example. Its routingConfiguration contains agentVersion, so an alias update can change which agent version a stable alias targets. Record the alias identity, affected version, responsible principal, and change event reference in the investigation. An event that updates only an alias description does not establish a routing change: inspect the actual configuration involved.

CreateGuardrailVersion creates a configuration snapshot. Include that event when reviewing policy changes, but distinguish version creation from use: a newly created version does not prove an affected request selected it. Join the event to the guardrail version recorded by your application for that request.

PutModelInvocationLoggingConfiguration changes loggingConfig, including delivery destinations and enabled data modalities. This event belongs in the timeline even when model behavior itself did not change, because it changes what evidence the investigation can observe. See 4.3.2 for the logging scope and destination configuration rather than inferring complete capture from the existence of a log group.

For each suspected transition, compare the last reviewed requests under the previous configuration with the first requests observed under the candidate configuration. A deployment timestamp marks an administrative action; it does not establish instantaneous adoption across in-flight requests or cached application configuration. Preserve overlap instead of forcing every request after the event into one release population.

Test behavior changes against release identity

Attach the resolved model target, prompt revision, generation configuration, retrieval-source revision, and policy revision to the request history. Preserve values or immutable artifact references rather than an environment label such as production. The timeline can contain several changes close together, so the actual dependency set is essential for attribution.

Use reviewed cases and acceptance rules to compare releases; see 4.3.9 for golden-dataset diagnosis. Retain case identifiers, candidate outputs, reviewed labels, and evaluator revisions with the release comparison. A changed answer might reflect a source update, changed user population, prompt release, or routing decision rather than a change in the model itself.

Start from the first observed unacceptable outcome and inspect preceding relevant management events. Then group requests by the configuration actually used. If the issue appears only under one agent version while source and policy revisions remain constant, prioritize that version for controlled reproduction. If both agent versions fail only after a source update, investigate the evidence boundary before changing the model. Grouping requests by the configuration actually used decides what to investigate first.

A correlation is an investigative lead, not proof of cause. Keep successful requests from the same period and request class as comparison evidence. Document which dependencies you held constant and which remain unknown. An investigation that preserves uncertainty is more defensible than attributing every behavior shift to the nearest configuration event.

Preserve evidence and capture gaps

Use the Object Lock retention and legal-hold controls in 3.3.4 for required evidence versions; masking is not deletion or minimization, so follow 3.2.9 for log and trace sanitization.

Include capture coverage in the case record: which request paths were logged, which modalities were enabled, and whether a relevant logging configuration changed during the incident. A missing payload after a logging change is an evidence gap. It does not establish that the model received no input or produced no output.

Retain the timeline, release comparison, and selected user-action history together with approved evidence references. This lets another investigator reproduce the reasoning without unrestricted transcript access. Protected storage preserves what was recorded; coverage review establishes which conclusions the available record can support.

Key distinctions and trade-offs

API audit versus business history: CloudTrail answers which AWS principal called an API. Your application answers which authenticated person requested a business action and whether it completed.

Behavior change versus model change: An outcome difference is an observation. Attribution requires matching configuration, evidence, and test conditions.

Protected evidence versus routine telemetry: Rich payloads improve diagnosis but broaden data exposure. Field selection and separate access paths let routine investigation proceed without unrestricted transcripts.

Immutable retention versus complete capture: S3 Object Lock protects recorded versions. It cannot recover an event that the application never emitted.

Architecture / decision scenario

A procurement assistant starts recommending an unapproved supplier. Investigators receive a complaint containing the application’s request identifier.

The request history shows the authenticated employee’s submission, the authorization result, two inference attempts, the delivered answer, and the user’s later selection. Correlated invocation logs establish the model inputs and outputs. CloudTrail confirms the application role’s API activity; the change timeline also shows an UpdateAgentAlias event shortly before the complaint. The backend confirmation record shows that no purchase completed.

Grouping histories by release identity shows that both agent versions fail only with a changed source revision. The alias update preceded the complaint but does not explain the shared failure. Investigators also verify that no PutModelInvocationLoggingConfiguration change created a capture gap. Reviewers compare the retained evidence with approved cases, correct the source, and validate the replacement. Sensitive raw evidence stays restricted; the investigation preserves the necessary S3 object versions under the approved retention policy.

Common exam traps

Treating a shared AWS role as the identity of the end user.
Treating the nearest alias update as proof of the cause of a behavior shift.
Treating CloudTrail as a full prompt-and-response transcript.
Calling an observed answer difference model drift without checking dependencies.
Treating guardrail version creation as proof that requests used that version.
Equating immutable storage with a complete request history.

What to remember

Join application requests, individual attempts, API events, and model evidence explicitly.
Record user intent, acceptance decisions, delivery, and business completion separately.
Track the configuration and source revisions actually used by each request.
Protect retained payloads with access controls and purpose-specific retention.