AWSAWS Certified Generative AI Developer - Professional2026-105.2 Troubleshoot GenAI applications
Locate context loss before changing models
Completed by the knowledge check
Learning objective
Diagnose context overflow and silent truncation with token accounting, budget allocation, and dynamic chunking.
Core idea
An assistant forgets a requirement halfway through a long conversation. The symptom does not tell you whether the model exceeded its context window, your application removed the requirement, or generation stopped before finishing the answer. Diagnose the boundary where information disappeared before increasing capacity.
Context overflow is a capacity failure. Silent truncation is information loss that your application might hide behind a successful invocation. A request that fits can still be missing the decisive paragraph. Keep evidence of what you intended to include, what you actually serialized, and why each reduction occurred. Conversation assembly is covered in 1.3.9; this lesson uses that assembly contract to localize faults.
How it works
Reconcile the actual request
Amazon Bedrock’s CountTokens operation returns inputTokens for the specified modelId. Its input union accepts converse for conversation input or invokeModel for a native request body. Tokenization is model-specific: a count for one model is not a capacity guarantee for another. Check the selected model’s token-counting support in models at a glance (model cards). Some Anthropic Claude models, including those launched with cross-Region inference only, do not support CountTokens on bedrock-runtime; do not assume that every invocation route provides this operation.
Count the assembled inference input. The converse member uses ConverseTokensRequest, whose fields are messages, system, toolConfig, and additionalModelRequestFields. Include the same tool definitions and supported additional parameters as the inference request. Counting the question alone misses conversation history and retrieved evidence. AWS states that the returned count matches the token count charged if you send the same input to the model, making it a diagnostic baseline. Do not replace model-specific accounting with a fixed characters-per-token rule.
Create an application diagnostic record with the selected model, request configuration revision, input count, output reservation, and retained evidence identifiers. Also record the history turns and source passages excluded by the assembler. These are application records, not automatic fields returned by CountTokens. Protect their content according to 3.2.9.
For successful Converse calls, inspect usage.inputTokens, usage.outputTokens, and usage.totalTokens, together with stopReason. Reconcile preflight and runtime counts using the same assembled input. A difference is a reason to investigate serialization, request changes, or model selection; it is not by itself proof that Bedrock silently discarded text.
Separate input capacity from output termination
Converse.stopReason includes model_context_window_exceeded, max_tokens, and stop_sequence. The first is an explicit context-capacity signal. max_tokens indicates output termination at a token limit, while stop_sequence identifies a configured stopping condition. Check these signals before diagnosing a missing closing paragraph as lost input.
inferenceConfig.maxTokens sets the maximum generated response length. It does not enlarge the model’s context capacity or restore removed history. Use the selected model’s documented input, output, and combined-context rules to establish an application budget. There is no single window size that applies to every Bedrock model.
Allocate room for mandatory instructions, the question, required conversation dependencies, retrieved evidence, and generation. Treat output reservation and accounting headroom as explicit design choices. A useful packing rule is:
verify model-specific capacity rules
reserve mandatory content and planned output
allocate remaining input space to useful evidence
reduce optional content, then recount the assembled request
if required evidence still cannot fit, split the task or decline itThe rule is an application policy, not a Bedrock parameter. Record which category exhausts its allocation. If mandatory instructions alone consume most of the available space, reducing retrieved passages treats the wrong cause.
Resize evidence without deleting its meaning
With dynamic chunking, you adapt the evidence admitted for one request to its remaining budget: select fewer passages, split an oversized passage at meaningful boundaries, or choose a narrower excerpt around the supporting clause. Keep the source identifier and retained boundaries so the diagnosis can prove which evidence reached the model.
Do not confuse request-time packing with Knowledge Bases ingestion chunking. Amazon Bedrock Knowledge Bases splits text during ingestion and writes embeddings to a vector index. Fixed-size chunking controls the maximum tokens and overlap for those chunks. Hierarchical chunking retrieves child chunks and can replace them with broader parents, so the assembled context can be larger than a child-chunk estimate suggests. See 1.5.1 for strategy selection and 1.5.11 for authorized context assembly.
Count the actual returned text after retrieval. Selecting five results is not the same as allocating a fixed token quantity. Deduplicate repeated passages, preserve exceptions beside their rules, and avoid cutting a table heading away from its values. When a smaller excerpt loses the answer, prefer fewer complete evidence units over many fragments.
Key distinctions and trade-offs
A token fit versus an evidence fit: Capacity accounting proves that a request can fit under the applicable rules. It does not prove that required facts survived selection. Track both token quantities and evidence identity.
Ingestion chunks versus request excerpts: Ingestion establishes searchable units. Request-time selection establishes what the generator receives. Changing ingestion is a broader intervention than narrowing one oversized context bundle.
Summarization versus preservation: A summary can reduce history, but it can also omit a condition or negate an exception. Test summaries against required facts rather than treating shorter text as an automatic repair.
More output versus more context: Increasing maxTokens can help an unfinished answer when output limits permit it. Under combined-context constraints, reserving more output also leaves less room for input. Apply the selected model’s rules instead of increasing both indiscriminately.
Architecture / decision scenario
A procurement assistant answers short questions correctly but omits cancellation exceptions after long exchanges. The application retrieves policy sections and includes prior approval messages.
The engineer reproduces one failure with the exact assembled request. CountTokens shows that the application’s planned allocation is exceeded. The assembler’s exclusion record reveals that it removes the oldest approval message and then slices the final policy passage. The returned Converse response has stopReason: end_turn; this does not establish that the omitted exception was ever available to the model.
The repair reserves the approval dependency and the complete cancellation clause before optional history. Request-time chunk selection excludes duplicate policy introductions and recounts the retained input. A second case finishes with max_tokens, so the engineer adjusts its output allowance separately within the model’s limits. An intentionally oversized control case exercises context-overflow handling rather than accepting an incomplete answer.
The team verifies all three cases: required facts survive packing, oversized requests follow the explicit failure path, and output-limited answers are distinguished from lost-input answers. It retains the failing long conversation as a regression case.
Common exam traps
- Counting only the user question instead of the assembled model input.
- Treating a successful HTTP response as proof that no application truncation occurred.
- Confusing
max_tokenswithmodel_context_window_exceeded. - Assuming a requested number of retrieval results establishes a token budget.
- Increasing output allowance without checking the model’s combined-context rules.
- Cutting evidence by character count while losing qualifiers and provenance.
What to remember
- Compare intended context, serialized context, and runtime accounting.
- Use model-specific
CountTokenswhere supported and inspectConverse.stopReason. - Reserve required meaning before optional history and retrieval content.
- Adapt request-time evidence to available space, then recount it.
- A complete answer needs both capacity and the decisive evidence.
Knowledge check
CountTokens shows that a long procurement request fits the model’s capacity, but the answer ignores an approval condition. The assembler’s exclusion record shows that it dropped the oldest approval message to stay within its allocation. Where did the information disappear?
A response is missing its closing paragraph, and Converse returned stopReason: max_tokens. What should the engineer investigate first?
max_tokens signals output termination at a token limit, not lost input; a capacity failure is reported separately as model_context_window_exceeded . Adjust the output allowance within the model’s rules — remembering that under a combined-context limit, more output leaves less room for input.An application counts only the user’s question with one model, then sends a different model a request containing tools, history and retrieved passages. Which preflight count is a useful baseline?
CountTokens is model-specific and returns the count that would be charged for the same input, so it must cover the whole assembled request — including toolConfig and system through ConverseTokensRequest — for the model that will actually run it.A context bundle is over budget, and cutting each passage to a fixed length removes the exception that decides the answer. Which packing repair preserves the evidence contract?
maxTokens so the answer has room for the exceptionAn assembler budgets five retrieval results at the 300-token child-chunk size of a knowledge base that uses hierarchical chunking. Assembled requests keep exceeding the planned input allocation. What explains the gap?
CountTokens does not count text from retrieved passages