Read runtime metrics within their boundaries

Learning objective

Track token consumption and invocation latency with available Amazon Bedrock and CloudWatch metrics.

Core idea

A metric becomes useful only when you know what it counts. Token totals describe workload consumption; invocation latency describes the model-call boundary; neither describes an entire customer request containing retrieval, retries, and tool execution.

Amazon Bedrock publishes bedrock-runtime inference metrics under the AWS/Bedrock CloudWatch namespace. Select the endpoint, model dimension, period, and statistic deliberately. Keep application-level duration and outcome measurements beside the service metrics so an improvement in one panel does not hide a regression elsewhere.

The monitoring decision is whether a change comes from workload growth, longer requests, slower generation, retries, or rejected traffic. Start with that question before selecting a graph.

How it works

Select the runtime metrics

For the runtime endpoint, these metrics establish the basic token and latency view:

TimeToFirstToken applies to ConverseStream and InvokeModelWithResponseStream. It is not a nonstreaming response-time metric. A streaming interface can deliver its first token quickly while taking much longer to finish; retain both latency views.

The runtime metrics page directs applications using bedrock-mantle to that endpoint’s separate metric documentation. Do not assume the runtime metric names and dimensions apply unchanged to another endpoint.

Use Sum over a common period to track token volume and accepted invocation count. Use an average to understand typical duration, and inspect latency percentiles to expose the slow tail where available. Keep the statistic visible in the widget label: a p95 latency and a mean latency answer different questions.

Partition the view by ModelId when comparing model-specific behavior. Also retain the dimensionless view for aggregate error accounting. Bedrock publishes runtime metrics with and without ModelId; errors occurring before model identification appear without that dimension. A dashboard containing only model-filtered error series can omit part of the failure picture.

Count accepted processing separately from rejection

Invocations counts requests to Converse, ConverseStream, InvokeModel, and InvokeModelWithResponseStream that Bedrock accepted for processing. It includes accepted requests that later fail. It excludes throttled requests and requests rejected during input validation before processing begins.

InvocationClientErrors and InvocationServerErrors describe their respective error classes and are also counted in Invocations. AWS explicitly distinguishes requests rejected for exceeding a service quota, which are client errors, from requests counted as throttles.

InvocationThrottles counts throttled invocations separately. They are not counted in Invocations, InvocationClientErrors, or InvocationServerErrors. SDK retry settings affect the number of throttles observed.

Do not add error counts to Invocations to calculate accepted processing; that double-counts errors already included. Do not label Invocations as successful customer requests either. A request may create several accepted model calls and several rejected attempts before returning one application outcome.

Maintain application counters for ingress requests, downstream attempts, terminal errors, and accepted business results. Their names and accounting contract are your implementation, not built-in Bedrock metrics. Correlate attempts as described in 4.3.1 and bound retries as described in 1.2.5.

Connect aggregates to per-call measurements

The Converse response includes usage.inputTokens, usage.outputTokens, usage.totalTokens, usage.cacheReadInputTokens, and usage.cacheWriteInputTokens. It also includes metrics.latencyMs. Record available response measurements against the individual attempt, while measuring the application’s own elapsed time around the call.

Application elapsed time can include client-side overhead and differs from the reported service measurement. End-to-end duration includes work outside inference. Keep those boundaries explicit rather than subtracting unrelated averages to invent retrieval or network time.

Per-call measurements let you compare like-for-like requests by model, prompt version, and workload class. Aggregate token growth might reflect more traffic or larger contexts. Compare token totals with accepted invocation volume, then inspect representative attempts. See 4.1.1 for spend attribution and 4.1.2 for token-reduction decisions.

Separate throughput from response length

AWS documents output tokens per second (OTPS) as a diagnostic for streaming inference. The calculation uses OutputTokenCount, InvocationLatency, and TimeToFirstToken: divide output tokens by the difference between completion latency and first-token latency, then multiply by 1,000 to convert milliseconds to seconds.

For an operational graph, the documented CloudWatch example uses aligned p50 statistics over five-minute periods. Label such a graph as an aggregate diagnostic. Combining aggregate values does not reconstruct the throughput of every individual request, especially when the window mixes workloads.

A longer output can increase completion latency while throughput stays stable. A throughput drop suggests a different investigation from a deliberate increase in response length. Use per-call records to confirm the explanation and avoid calculating a ratio when its denominator is missing or nonpositive.

EstimatedTPMQuotaUsage is approximate and does not reflect the reservation-based token accounting that drives throttling. AWS describes upfront reservation of input tokens plus the requested maximum output tokens. Cache-read tokens do not count toward the tokens-per-minute quota; cache-write tokens do. Do not use the estimate alone to declare spare capacity.

Key distinctions and trade-offs

Volume versus efficiency: A lower token total can result from fewer requests. Compare volume, per-attempt usage, and accepted outcomes before declaring an optimization successful.

First token versus completion: Initial streaming responsiveness and complete-answer duration need separate targets.

Consumption versus quota versus bill: Observed usage, reservation-based throttling, and billed cost have different accounting boundaries. The same token chart cannot establish all three.

Model-level dimensions make comparisons clearer but can hide pre-model errors. An aggregate view avoids that omission but mixes models. Use both, then drill into the appropriate cohort.

Architecture / decision scenario

A support workload shows a latency increase after a prompt release. InputTokenCount and OutputTokenCount increase, but TimeToFirstToken changes little. The streaming OTPS diagnostic stays near its baseline.

Per-call Converse records confirm that the changed configuration produces longer answers. The team investigates response-length policy rather than assuming service throughput has degraded. It checks accepted-response quality before reducing output.

A second panel shows growing InvocationThrottles. The operator inspects attempt records and retry behavior separately from accepted Invocations. The dimensionless error view also exposes failures missing from model-filtered charts. This produces two bounded investigations instead of one misleading average.

Common exam traps

• Calling Invocations a count of successful user requests.
• Adding client and server errors to accepted invocation counts.
• Expecting TimeToFirstToken for nonstreaming calls.
• Filtering all errors by ModelId and losing pre-model failures.
• Treating longer completion latency as proof of slower token generation.
• Using estimated quota usage as an exact throttling or billing ledger.

What to remember

• Read token volume, latency, and errors with explicit metric boundaries.
• Use both model-specific and dimensionless views.
• Separate accepted processing, throttles, retries, and business outcomes.
• Compare first-token and completion latency for streaming traffic.
• Confirm aggregate explanations with per-attempt measurements.

Knowledge check

References

• AWS, Monitor bedrock-runtime inference using CloudWatch metrics
• AWS, Converse
• AWS, Diagnose InvocationLatency increases using output tokens per second
• AWS, CloudWatch statistics definitions