AWSAWS Certified Generative AI Developer - Professional2026-100 Prerequisites
Budget context before discarding input
Completed by the knowledge check
Learning objective
Understand tokenization, context windows, and truncation.
Core idea
When you assemble a model request, the resource boundary is the model’s token budget, not the document’s page count. A token is a unit the model processes or predicts. It can represent a word, part of a word, punctuation, or another character sequence. Tokenization converts content into those units, and different models can tokenize the same input differently.
A context window bounds what the model can consider for an invocation. A separate output limit bounds what it can generate. Truncation removes content because a boundary is reached or because the application deliberately reduces input. These are distinct events with different consequences. Fitting the request is necessary, but removing the sentence that determines the answer can make a fitting request incorrect. See 1.3.9 for conversation assembly and 4.1.2 for reduction policy.
How it works
Count the request the model actually receives
Do not equate words with tokens. A document with product codes, punctuation, source code, or another language need not share the token count of an English paragraph of the same length. Character-based estimates are useful for early sizing, but they are not an exact admission check for a model request.
Amazon Bedrock provides CountTokens for supported models. You specify modelId and an input matching either the invokeModel or converse input representation. The response’s inputTokens gives the model-specific count. Count the final assembled request, including instructions and supplied conversation context, rather than counting only the customer’s current sentence. A retrieved passage that enters the prompt consumes input capacity even if it did not originate with the customer.
AWS documents an important support boundary: some Anthropic Claude models do not support CountTokens on bedrock-runtime. For those models, the token-counting guide directs you to Anthropic’s count_tokens API on bedrock-mantle. Check the selected model’s support rather than treating one counting operation as universal. A counting failure is not evidence that a request fits.
Token counting tells you about supplied input. It does not predict the exact length of a response that has not yet been generated. After inference, Converse.usage.inputTokens, outputTokens, and totalTokens describe actual usage. Compare the estimate and actual consumption for the same request path, especially after changing models or input structure.
Separate the context window from output capacity
The context window is the amount of information the model can consider, expressed in tokens. It is not a promise that the model will reproduce every detail or answer every question about that information. A long request can fit while still producing a poor answer because its instructions or evidence are ambiguous.
The model’s maximum generated output is a different constraint. AWS documents Amazon Nova Micro with a 128K context window and a 10K maximum output. Those numbers do not imply that you can request a 128K response. They describe different boundaries for that particular model. Consult model-specific documentation for how input, generation, and any additional processing budgets interact; do not apply one family’s accounting rule to another.
In Converse requests, inferenceConfig.maxTokens caps generated response tokens. This setting does not enlarge the context window or remove excess input. The API schema accepts a positive integer, but the selected model determines the usable maximum. Increasing the requested output beyond that model’s supported limit is not a way to obtain a longer answer.
Use an explicit planning budget: allocate space for required instructions, necessary evidence, the current request, and the intended output under the selected model’s rules. If something must shrink, decide which information is redundant before removing it. The exact accounting belongs to the chosen model; the application still owns what meaning it is willing to lose.
Distinguish input reduction from output truncation
Input reduction happens before invocation. You might remove repeated passages, select only relevant sections, or replace earlier discussion with a reviewed summary. Every reduction changes what the model can use. Preserve qualifications, exceptions, identifiers, and dependencies that determine the requested decision. A summary is another representation to validate, not equivalent evidence by definition.
Do not assume Bedrock silently discards the oldest messages whenever an input is too large. Verify the request contract and reject or reduce oversized input deliberately. If you send the whole conversation, the complete representation is what needs budgeting. If you omit earlier context, the model cannot rely on that omitted context merely because the application displayed it to the customer.
Input truncation can also be a model-specific request setting. Cohere Embed v3’s truncate accepts NONE (error on oversized input), START (discard the start), and END (the default, discard the end); START and END discard content until the remaining input reaches the model’s maximum input token length. Set NONE when silent content loss is unacceptable, and handle the error through deliberate input selection rather than relying on the default.
Output truncation happens during generation. Converse returns stopReason; max_tokens means the output-token limit stopped generation. Treat a cut-off sentence or incomplete structured result as an incomplete candidate, even when the HTTP request succeeded. The API also documents model_context_window_exceeded as a possible stop reason. It is distinct from max_tokens and should remain distinct in application diagnostics.
Input is reduced before invocation, while output truncation is read from stopReason after inference.
Key distinctions and trade-offs
Characters versus tokens: Characters describe the input encoding. Tokens describe model processing units. The conversion depends on the model and content.
Input budget versus output cap: Reducing maxTokens constrains generation. It does not automatically shorten the supplied document or make missing evidence available.
Truncation versus selection: Cutting at an arbitrary boundary saves space but can remove an exception halfway through a rule. Selecting complete relevant passages makes the retained evidence easier to inspect, although selection can still miss required information.
Fit versus answer quality: A larger window offers more capacity. It does not establish that including every available passage improves the answer. Capacity and representative quality tests answer different questions.
Architecture / decision scenario
A support assistant receives a long troubleshooting manual, an ongoing conversation, and a question about whether a warranty exception applies. The first implementation counts only the last question and requests an output as large as the model’s context window.
The corrected design budgets the complete request. It uses the supported model-specific token-counting path and compares inputTokens with the selected model’s documented limits and planned generation allowance. It recognizes that Nova Micro’s 128K context and 10K output maximum are separate constraints.
If the manual does not fit, the application selects the warranty rule and its exceptions together, rather than cutting off the manual at a character limit. It keeps the product version and the relevant customer facts. The generated answer is accepted only after inspecting stopReason and validating the result. A max_tokens response does not become a completed warranty decision; the application can request a shorter answer or route the case for review. The scenario fixes token admission, evidence preservation, and output completion as three separate responsibilities.
Common exam traps
- Assuming one word always equals one token.
- Reusing a token count after changing the model or assembled input.
- Treating context-window size as maximum response length.
- Assuming
CountTokenspredicts future output tokens. - Assuming oversized conversation history is safely truncated automatically.
- Assuming oversized embedding input always fails instead of following a model-specific truncation default.
- Treating
stopReason: max_tokensas a fully completed answer.
What to remember
Tokenization is model-specific. Count the complete intended input through a supported mechanism, budget generation separately, and preserve the evidence that determines the answer. Inspect completion metadata because fitting input and successful transport do not prove that the generated result is complete.
Knowledge check
A team moves an assistant to a different model and keeps the same instructions, conversation history and retrieved passages. How should it confirm the new request fits the input limit?
A model has a 128K context window and a 10K maximum output. A product owner asks for 20K-token reports and proposes setting maxTokens to 20,000 in the Converse request. What is the problem with this plan?
maxTokens saysmaxTokens caps generation within what the selected model supports — it neither enlarges that limit nor trims the input.A request fits the model’s input limits and returns HTTP 200, but its JSON response ends mid-record with stopReason set to max_tokens. What should the application do?
max_tokens means the output-token limit stopped generation, so the result is incomplete even though the input fit and the transport succeeded. It is also distinct from model_context_window_exceeded ; trimming input does not address an output cap.An indexing job embeds support articles with Cohere Embed v3 on Amazon Bedrock. Losing the end of a long article without notice would hide policy exceptions from search. How should the job set truncate?
NONEENDSTARTtruncate unset, the default END discards the end of oversized input until it fits, so content disappears silently. NONE returns an error instead, and the job can then select or split the content deliberately rather than lose an exception at an arbitrary boundary.