Preserve conversation meaning within context budgets

Learning objective

Assemble and truncate ordered conversation history within the model context budget.

Core idea

Conversation history is an ordered set of messages and dependencies. A context policy must preserve the information needed for the next answer while fitting the selected model’s documented limits. Dropping arbitrary characters until the request looks short can destroy roles, evidence, tool exchanges, or a signed reasoning sequence.

Bedrock’s documented Converse pattern includes previous messages in subsequent requests. Your application assembles that history. A model invocation is not a durable conversation store that automatically recalls every earlier request.

Separate storage from the inference view. Retain the authoritative conversation under your application’s storage policy, then construct the bounded view needed for one invocation. A reduced view should be explainable: which turns remain, which facts a summary represents, and which source evidence the next answer can still use.

How it works

Assemble ordered, complete units

Build messages in conversational order. After a successful Converse response, its output.message is the assistant message used in the documented continuation pattern. Keep structured content rather than replacing every assistant response with a concatenated text string; responses can contain more than text.

Keep application instructions in your chosen system representation and include the current user request. Add only the prior turns and evidence needed for that request, subject to the application’s policy. The syntax of messages and content blocks is covered in 1.3.6.

Treat related messages as reduction units. A user clarification and the answer it resolves are a stronger unit than either turn alone. A tool-use message and its result also have a relationship: ToolResultBlock.toolUseId identifies the tool request that produced the result. A retained result without its corresponding request loses the context for interpreting it.

This unit policy is application design, not an automatic Bedrock truncation feature. Define what your reducer is allowed to discard, and validate the assembled messages after each reduction. Do not rely on the model to reconstruct missing dependencies from a fragment.

Count the actual model input

The Bedrock Runtime CountTokens operation returns inputTokens for a request compatible with the specified foundation-model modelId. Tokenization is model-specific: the same source text can have different counts for different models. A character count or another model’s tokenizer is not the authoritative measurement for this operation.

For a supported Converse request, supply messages and system content under input.converse. For a native request, supply its body under input.invokeModel. These are alternatives in a union, not fields to combine in one counting request.

The following fragment illustrates the counting input for a text conversation:

{
  "input": {
    "converse": {
      "system": [{ "text": "Answer from supplied evidence." }],
      "messages": [{ "role": "user", "content": [{ "text": "Summarize the decision." }] }]
    }
  }
}

Supply the supported foundation model separately as modelId. Count the assembled content you intend to send, not an earlier draft that omits evidence or system instructions. Check the counting API’s supported input surface before relying on it for requests with additional features.

AWS documents CountTokens as not incurring charges, but support varies by model and Region. Some Claude models offered only through cross-Region inference on bedrock-runtime do not support its CountTokens operation. AWS documents Anthropic’s counting API at /anthropic/v1/messages/count_tokens on the bedrock-mantle endpoint for such models. That endpoint uses a different request shape and is not reached by the bedrock-runtime client’s count_tokens method.

Choose a supported counting path as part of model configuration. Do not assume that an inference-profile identifier accepted by Converse is accepted as a foundation-model identifier by CountTokens. If exact counting is unavailable in your integration, mark the estimate as an estimate and use a conservative policy rather than promising exact fit.

Reserve room for generation

Obtain the context and output limits from the selected model’s documentation; see 1.2.1. Where the model’s context limit accounts for input and generated tokens together, reserve the intended output budget before approving the input. Also enforce any separately documented input or output ceiling.

For Converse, inferenceConfig.maxTokens sets the maximum generated output. It is not an instruction to prune the input. A practical application policy compares measured input, reserved output, and a chosen safety margin against the applicable model limits. The margin is an application choice, not an AWS quota.

For example, suppose an application has verified a 32,000-token combined context limit for its selected configuration. It reserves 2,000 tokens for output and 1,000 for its safety margin. Its input gate is then 29,000 tokens. These are illustrative planning values, not claims about a named Bedrock model.

Reduce without violating immutable dependencies

First remove irrelevant optional material. If more reduction is necessary, drop complete earlier exchanges or replace suitable history with a reviewed, source-traceable summary. Retain the current request and the facts required to interpret it. Count again after every change because a summary has its own token cost.

A summary is transformed data. It can omit an exception, merge speakers, or convert uncertainty into certainty. Validate important commitments against the original messages and retain identifiers that let the application recover evidence. The validation boundary for generated transformations is covered in 1.3.10.

Signed reasoning content requires a different path. AWS documents that reasoningContent.reasoningText.signature is a safeguard tied to the conversation messages. Subsequent Converse requests must include the signature and all previous messages; changing a message causes an error. Do not apply ordinary truncation to such a continuation.

If that intact continuation cannot fit, stop that continuation and use a separately constructed request or a product fallback. Do not splice the signature onto a rewritten transcript and claim continuity. The application must preserve any required business facts independently.

The reducer loops back to counting until the view fits the input gate, while a signed continuation that does not fit leaves for a separately constructed request.

Key distinctions and trade-offs

Stored history versus sent context: Retaining a transcript does not mean sending it all on every request. The inference view can be bounded while the original remains available under the storage policy.

Truncation versus summarization: Dropping exchanges loses them explicitly. Summarization keeps a derived representation but introduces transformation risk. Neither guarantees that every earlier qualification remains available to the model.

Input fit versus answer completion: A counted input can fit while generated output reaches its cap. Check Converse stopReason, including max_tokens or model_context_window_exceeded, alongside usage.inputTokens and usage.outputTokens.

Editable history versus signed continuation: Ordinary message reduction is an application policy. Signed reasoning imposes an integrity constraint that takes precedence over that reducer.

Architecture / decision scenario

A support assistant accumulates customer questions, assistant answers, and diagnostic tool results. Before the next invocation, it constructs an ordered view containing the current problem, relevant recent exchanges, and matched tool requests and results.

Its supported counting path reports that the input exceeds the configured gate. The reducer removes an irrelevant resolved exchange, then summarizes an earlier discussion of the affected product version. It checks that summary against the original turns and counts the final view again before calling Converse.

In a signed reasoning continuation, the same reducer is disabled. If the intact messages cannot fit the budget, the application creates a separate request with validated business state instead of altering signed history. It records the loss of conversational continuity explicitly.

After inference, the application records token usage and completion status. A capped answer does not become a complete diagnostic record merely because the input passed counting.

Common exam traps

• Assuming Bedrock automatically remembers earlier Converse requests.
• Counting characters instead of tokens for the selected model.
• Setting maxTokens and expecting input history to be truncated.
• Keeping a tool result while dropping its corresponding request.
• Assuming CountTokens supports every inference model, Region, or identifier.
• Editing earlier messages in a signed reasoning continuation.
• Treating a generated summary as an authoritative original transcript.

What to remember

• Assemble structured messages in their original order.
• Measure the final input with a supported, model-specific counting path.
• Reserve output capacity and enforce all applicable model limits.
• Reduce complete units and validate summaries against original evidence.
• Preserve signed reasoning history intact or choose a separate request path.

Knowledge check

References

• AWS, Use the Converse API
• AWS, CountTokens
• AWS, Count tokens before you run inference
• AWS, ToolResultBlock
• AWS, Converse