AWSAWS Certified Generative AI Developer - Professional2026-104.2 Optimize application performance

Locate delay on the request path

Completed by the knowledge check

Learning objective

Profile FM API calls, vector queries, and network boundaries to break down time to first token, generation, retrieval, orchestration, and network latency.

For an architect diagnosing a slow assistant, the outcome is a timing model that identifies which boundary delays useful output. This lesson focuses on attribution; see 2.4.9 for failure correlation and 4.2.2 for latency reduction.

Core idea

A slow answer does not imply slow foundation model (FM) inference. A request can wait for authorization, query embedding, vector search, tool calls, retries, or delivery buffering before the reader receives anything. Optimizing a stage outside the critical path might reduce resource use without changing the user experience.

Measure two endpoints: the first useful response content and the completed response. Keep application timings separate from service metrics. Amazon Bedrock’s TimeToFirstToken describes a streaming model invocation, not the elapsed time from a user submitting a question through retrieval and orchestration to a browser displaying text.

Build a timeline of individual attempts, including failed attempts, rather than a single duration labeled “AI latency.” Then select the stage whose improvement can actually move the endpoint you care about.

How it works

Establish timing boundaries

Record application timestamps for request acceptance, retrieval start and end, model request start, first model content, final model content, and client display. Use a monotonic clock for elapsed durations inside one process. Across processes, correlate tracing context and inspect clock alignment before subtracting timestamps.

Define first useful content explicitly. A stream connection, a role event, or a progress indicator is not necessarily an answer token. In ConverseStream, contentBlockDelta carries content changes, while messageStop includes stopReason and metadata carries usage and performance information. Separate text from tool-use content when measuring a text-answer experience.

Preserve the selected model, request shape, input and output token counts, attempt number, and whether the request streams. Avoid recording prompt bodies merely to measure elapsed time. See 3.2.9 for sanitizing telemetry.

Profile the FM call

The Bedrock runtime endpoint publishes metrics in the AWS/Bedrock namespace. InvocationLatency measures the interval from sending a request to receiving the last token. TimeToFirstToken measures the interval to the first token for ConverseStream and InvokeModelWithResponseStream. It is not a metric for every inference API.

For a stream, the interval between first and last token isolates the visible generation period in your application measurement. It does not prove that every millisecond in that interval is model computation: stream transport and consumption can also contribute. Record metadata.metrics.latencyMs and metadata.usage.inputTokens and outputTokens from ConverseStream alongside your timings.

AWS decomposes model-host work into prefill and decode. Prefill processes the input and drives time to first token, scaling primarily with input length. Decode generates subsequent tokens, so its duration scales with output length. The documented relationship is InvocationLatency (ms) = TimeToFirstToken (ms) + (OutputTokenCount / OTPS) * 1000, where OTPS is output tokens per second.

Compute OTPS with CloudWatch metric math: OutputTokenCount / (InvocationLatency - TimeToFirstToken) * 1000. Use those three metrics from the AWS/Bedrock namespace for comparable traffic. This calculation applies only to ConverseStream and InvokeModelWithResponseStream, because it requires TimeToFirstToken. Stable OTPS indicates expected generation throughput even when longer prompts or responses raise invocation latency; a drop indicates a model-side throughput change.

Include InvocationThrottles, InvocationClientErrors, and InvocationServerErrors in the investigation. An application request that succeeds after retries still contains the wait and work of those attempts. A fleet-level successful-invocation latency cannot explain that complete request by itself.

Separate retrieval from generation

Time query preprocessing, query embedding, the vector-store round trip, and post-retrieval processing as separate application stages where your architecture exposes them. An embedding request is not the vector query: Titan Text Embeddings generates the representation; the vector database performs similarity computation and retrieval.

With Bedrock Knowledge Bases, Retrieve exposes a retrieval-only operation. Time the Retrieve call and retain the result count and source identifiers from retrievalResults for comparison. If the response includes nextToken, account for any subsequent page requests rather than silently comparing one-page and multi-page workloads.

A managed retrieval call remains a boundary measurement. Do not label its entire duration “vector-index execution” when it also includes work that the service does not expose separately. For a custom store client, create a dedicated vector-query span; for a managed boundary, report the available evidence and its limits.

Trace orchestration and network edges

AWS X-Ray represents a request as a trace of segments and subsegments. A downstream client subsegment records the caller’s view of a call, while an instrumented downstream segment records the service’s own work. AWS documents that the caller’s round trip includes travel between the services.

Use OpenTelemetry instrumentation, which AWS recommends, and export spans to X-Ray through AWS Distro for OpenTelemetry. Add explicit spans for query preparation, vector calls, model calls, result assembly, and delivery. Propagate context across service boundaries so the timeline shows dependencies rather than disconnected durations.

Treat a gap between caller and server timing as a clue, not a pure network measurement. It can include connection setup, intermediary handling, queueing, or clock differences. Instrument exposed HTTP boundaries and record client receive timing to narrow the cause.

Key distinctions and trade-offs

Model TTFT versus application TTFT: Bedrock TimeToFirstToken measures a model-call boundary. Application TTFT includes all prerequisites and downstream delivery.

Generation versus completion: The first-to-last-content interval describes generation and delivery after initial content. End-to-end completion also includes preprocessing, retrieval, and orchestration.

Stage totals versus critical path: Concurrent spans overlap. Adding every duration double-counts parallel work. Follow the dependency chain that determines the response endpoint.

Metrics versus traces: CloudWatch metrics show distributions and trends. A trace shows the work and dependencies of a particular request. Use both, and avoid treating a sampled trace as the population.

Compare p50 and p95 application timings by request class, token size, and execution route. A fleet average can hide a retrieval tail affecting only permission-heavy requests. Additional instrumentation has collection and processing cost, so keep fields targeted to the decision.

Architecture / decision scenario

A support assistant retrieves documents, checks a product entitlement, and streams a Bedrock answer. Users report a long initial pause, but TimeToFirstToken for the model is stable.

The team creates OpenTelemetry spans for Retrieve, entitlement lookup, ConverseStream, and response delivery. The timeline shows retrieval and entitlement running sequentially before model invocation. A custom vector-query span reveals that the vector round trip is only part of retrieval; query preparation accounts for the rest.

The application also records the first text delta and the first client display. Some requests show a delivery gap after the model emits content, exposing a buffering boundary. Others spend their time before inference. The team separates those cohorts instead of choosing one universal fix.

They prioritize the serial prerequisites and delivery buffer. They retain token counts to verify that reduced completion latency does not merely reflect shorter answers.

Common exam traps

Calling end-to-end request latency TimeToFirstToken without its Bedrock scope.
Assigning the whole Retrieve duration to the vector index.
Adding overlapping spans to calculate wall-clock response time.
Treating the first stream event as useful answer text.
Alarming on InvocationLatency when output length grows without checking OTPS.
Ignoring retry attempts because the final model invocation succeeds.
Attributing caller/server timing differences entirely to network travel.

What to remember

Define first useful content and completion separately.
Use InvocationLatency, TimeToFirstToken, token counts, and per-attempt timings together.
Distinguish embedding, vector retrieval, orchestration, and client delivery.
Trace dependencies and optimize the critical path.
State what a managed service boundary does not reveal.