AWSAWS Certified Generative AI Developer - Professional2026-104.2 Optimize application performance
Prove sampling changes improve accepted responses
Completed by the knowledge check
Learning objective
Tune model-specific temperature, top-k, and top-p and confirm improvements with controlled A/B testing
Core idea
Sampling parameters change how a model chooses its next token. They do not add evidence, enforce authorization, or guarantee factual correctness. A lower temperature can make an unsupported answer more consistent; that is not an improvement in answer quality.
Choose a task-level hypothesis before choosing a parameter. For extraction, you might want fewer inconsistent field values without losing supported details. For drafting, you might want more useful alternatives without increasing rejection rates. A performance claim must also survive checks on response length, completion validity, latency, and token use.
The winning configuration is specific to the model version, task, and operating conditions. Use controlled A/B comparisons to establish that the change improves accepted results. Prompt refinement belongs to 1.6.7; this lesson isolates sampling changes so their effects remain interpretable.
How it works
Distinguish the three sampling controls
Temperature changes the shape of the next-token probability distribution. Lower values favor higher-probability choices; higher values make lower-probability choices more likely. It changes relative probabilities rather than specifying a fixed count of candidates.
Top-k restricts sampling to a count of the most likely next tokens. With a value of 50, the candidate pool contains the 50 most probable tokens. Top-p, or nucleus sampling, restricts the pool by cumulative probability mass. With a value of 0.8, the pool covers the most probable tokens up to that cumulative probability. Its size varies with the distribution.
These controls can interact, but their availability, ranges, and defaults depend on the model. Do not assume that a Bedrock-wide parameter description means every model accepts every control. Nor should you translate a useful numeric setting between model families and expect identical behavior: the distributions and tokenizer differ.
Use the supported request contract
For direct model calls through Converse, common controls live in inferenceConfig: temperature, topP, and the separate output limit maxTokens. The InferenceConfiguration API schema gives temperature and topP a range of 0 to 1, while defaults come from the selected model. A schema range is an interface boundary, not an instruction to sweep every value for every model.
Top-k is not a base Converse field. When the model supports it, pass the native field through additionalModelRequestFields; the AWS Converse example uses additionalModelRequestFields.top_k. Native Anthropic Claude Messages requests use temperature, top_p, and top_k. AWS documents Claude Messages top-k from 0 to 500 and disabled by default. Check the chosen model’s specific restrictions before sending it.
There is a concrete incompatibility to preserve: AWS states that Claude Sonnet 4.5 and Claude Haiku 4.5 accept either temperature or top_p, not both. The Claude guidance also recommends modifying temperature or top-p separately. Make one control the experimental variable and omit the incompatible field; explicitly sending its default still sends it.
A Prompt management version ARN changes where configuration belongs. In that Converse mode, you cannot include inferenceConfig or additionalModelRequestFields in the runtime request; define them through Prompt management instead. See 1.6.2 for prompt-version semantics. The experiment must identify the effective settings, whether they come from the direct request or the versioned prompt.
Establish a controlled baseline
Freeze the model identifier, prompt, retrieval configuration, corpus revision, tools, safety policy, output limit, and evaluation rubric. Record the baseline’s explicit and omitted parameters. An omitted field inherits the model default, so an experiment record containing only the changed value is incomplete.
Start with a representative offline test set and run both configurations against the same inputs. Separate task classes and include long prompts, difficult examples, and requests that should produce a bounded refusal or abstention. Repeat stochastic generations so one lucky response cannot determine the release. Use a held-out set to check a candidate chosen after several exploratory sweeps.
For the first comparison, change one supported sampling control. Hold the others fixed or omitted according to the model contract. If a later candidate combines changes, treat the combination as a separate configuration and compare it with the baseline. Do not claim that one parameter caused an effect when several changed together.
Compare quality and operating outcomes
Define the acceptance rubric before reading outputs. For extraction, check required fields against reviewed evidence. For drafting, use reviewed usefulness and factual-support criteria. Blind reviewers to the variant where practical. Select a minimum useful improvement and an uncertainty rule before promotion, rather than stopping after a favorable handful of responses.
Capture Converse usage.inputTokens, usage.outputTokens, metrics.latencyMs, and stopReason alongside the configuration and evaluated outcome. A response ending with stopReason: max_tokens might be shorter because it was truncated, not because the model became more efficient. Distinguish a legitimate completed answer from an incomplete result before comparing cost or speed.
After offline gates pass, carry each treatment’s sampling settings as an AWS AppConfig multi-variant feature flag value. Assign eligible traffic with a split rule keyed on the conversation ID supplied as request context. The operator uses a consistent hash of that value to select a percentage of traffic, keeping a conversation in the same variant while the rule and context remain unchanged. Use the baseline as the default variant and freeze the experiment’s rules during measurement. Keep traffic slices comparable, including prompt length and workload class. The statistical promotion decision remains an application responsibility; see 1.2.3 for AppConfig release mechanics.
Key distinctions and trade-offs
Consistency versus correctness: Less variation is useful only when the stable answer meets the acceptance criteria. Sampling does not repair missing retrieval evidence.
Sampling versus output limits: Temperature, top-k, and top-p influence token selection. maxTokens limits output length. Changing the latter during a sampling experiment confounds completion validity and token comparisons.
Offline comparison versus production A/B testing: Paired offline inputs isolate response differences. Randomized live assignment tests the effect under production conditions. Both need recorded configuration and predefined quality gates.
More diversity can produce useful alternatives but increase review burden. Lower diversity can improve format consistency while suppressing acceptable alternatives. Evaluate those trade-offs by task slice rather than choosing one setting for every request.
Architecture / decision scenario
A service extracts evidence-backed ticket fields and drafts optional customer replies. The team first checks each selected model’s API contract. It tests extraction with lower temperature while keeping top-p omitted on Claude Sonnet 4.5. For reply drafting, it compares a top-p treatment with temperature omitted; a separate top-k trial uses additionalModelRequestFields.top_k only on a compatible model.
The team freezes each route’s prompt, evidence, output cap, and validation rubric. Paired offline runs measure field accuracy or reply acceptance, token use, and latency. Repeated outputs reveal whether an apparent gain persists across generation variation.
Only candidates that pass the offline gates enter a live comparison. The team stores baseline and candidate sampling settings in an AppConfig multi-variant flag and selects candidate traffic with a split rule using the conversation ID as context. It records the returned variant and effective configuration with stopReason and usage, then applies the predefined statistical promotion gate. A candidate that reduces latency by cutting off valid replies fails the release gate. A candidate that improves accepted replies without harming factual support or operating limits can be promoted with its baseline retained for rollback.
Common exam traps
- Treating temperature zero as a factual-correctness guarantee.
- Confusing a fixed top-k count with top-p probability mass.
- Putting
top_kdirectly in ConverseinferenceConfig. - Sending temperature and top-p together to a model that prohibits the pair.
- Changing prompts, output limits, and sampling together, then attributing gains to sampling.
- Promoting a variant from one favorable output or unbalanced live traffic.
What to remember
- Check support and restrictions for the exact model and request interface.
- Temperature reshapes probabilities; top-k and top-p restrict candidate pools differently.
- Change one control first and freeze other inputs.
- Compare accepted quality, token use, latency, and completion reasons.
- Confirm offline gains through controlled production assignment before promotion.
Knowledge check
For a drafting task, a team wants the sampling pool to stay small when the model is confident about the next token and to widen when many tokens are similarly likely. Which control matches that behavior?
maxTokens , because it bounds how far generation can wandermaxTokens limits output length, not token selection.A team calls a model directly through Converse and wants to test top-k, which the model supports natively. Where does the setting go?
inferenceConfig , as a topK field next to topPadditionalModelRequestFields , using the model’s native field nameinferenceConfig , as a top_k field next to temperatureinferenceConfig carries the common controls such as temperature , topP and maxTokens ; top-k is not one of them. A supported model-specific field goes through additionalModelRequestFields , for example top_k , after checking the exact model’s parameters.An experiment on Claude Sonnet 4.5 changes temperature and keeps topP in every request at the value the team believes is the default. Requests are rejected. What should the treatment do?
topP as 1.0 so that it has no effecttopP into additionalModelRequestFieldstopP and vary only temperaturetemperature or top_p , not both. Sending a field at its default value still sends it, so the experiment keeps one control as the variable and leaves the other out of the request.A sampling A/B test shows a better acceptance rate for the treatment, which also used a revised prompt, a newer corpus revision and a larger output cap. What can the team conclude about the lower temperature?
A lower-temperature candidate shows 20% lower latency and fewer output tokens. Many of its responses end with stopReason set to max_tokens before required fields appear. What should the release decision be?
max_tokens stop means the response was cut off, so the savings come from incomplete answers rather than efficiency. Completion validity and task quality are gated first; speed and token use are compared only among accepted results.