Budget tokens before tuning model behavior

Learning objective

Relate tokens, context windows, inference parameters, and model limits to behavior, latency, and cost

Core idea

A model request spends a finite token budget and processing time. Tokens are the units the model processes, rather than a count of words or characters. The context window bounds what one request can contain; output limits bound generation; deployment quotas govern request admission. These are different constraints and require different remedies.

For an architect, tuning starts with a valid request and a measurable outcome. A large context window doesn’t guarantee a complete answer, a low sampling temperature doesn’t guarantee correctness, and streaming doesn’t reduce all of the work needed to finish. Record token usage together with response status and latency so that you can distinguish an output-budget failure from a capacity problem.

How it works

Count the whole request

Azure OpenAI converts input text into tokens before inference. Count instructions, the user’s request, conversation history, retrieved passages, and tool definitions. A tool schema is part of the prompt budget even when the model doesn’t invoke it. Tool results added on later turns enlarge the next request as well.

For Python Chat Completions integrations, Microsoft’s conversation-management example uses tiktoken.encoding_for_model to estimate token counts. Use an encoding that supports the deployed model and leave a safety margin; the example’s message accounting isn’t a universal formula for every model and request shape. Validate the estimate against the service’s reported usage.

Chat Completions reports usage.prompt_tokens, usage.completion_tokens, and usage.total_tokens. Reasoning detail appears in completion_tokens_details.reasoning_tokens; Responses reports output_tokens_details.reasoning_tokens. These fields explain why a short visible answer can consume substantial generation. They are evidence about a completed request, not an exact prediction for the next one.

Reserve context for generation

The model’s input, output, and reasoning tokens share the available context budget. Separate maximum-input and maximum-output figures aren’t additive entitlements. A large input leaves less room for generation, and setting a generous output cap doesn’t reserve unavailable space.

Microsoft documents an important API-specific example for gpt-5.5: its model context window is 1,050,000 tokens, but the Responses API’s effective combined prompt-and-generation budget is approximately 922,000 tokens. With 921,549 prompt tokens, only 451 remain for generation, including reasoning. Setting max_output_tokens to 128,000 doesn’t make that request produce 128,000 tokens.

That number belongs to the documented GPT-5.5 Responses implementation. Don’t copy it into every model adapter. Deployment type can also change the applicable limit; the model catalog distinguishes GPT-4.1 series context limits by deployment category. Keep the model, version, API, and deployment type beside each configured budget.

For multistep work, retain the evidence needed for the next decision rather than appending every observation indefinitely. See 0.3 for context selection. History trimming changes what the model knows, so treat it as a quality decision as well as a space optimization.

Match inference controls to the model

Sampling controls influence variation. For models that support temperature, Microsoft’s prompt-engineering guidance gives a range of 0 to 2: higher values produce more divergent responses, while lower values focus output. top_p is another sampling control. The guidance recommends changing one of the two at a time so you can attribute the effect.

Lower randomness doesn’t turn generation into deterministic business logic. Validate facts and application constraints independently. If two outputs differ only in wording, an exact string comparison might also misrepresent task success; choose a result contract that captures what matters.

Parameter support is model-specific. Don’t send temperature or top_p to a reasoning deployment unless its model/API documentation establishes support. Azure OpenAI’s reasoning guide names reasoning_effort for controlling reasoning work and describes supported values as model-dependent. A higher effort generally produces more reasoning tokens and takes longer. Compare accepted task results before increasing it.

For reasoning generation, use max_output_tokens with Responses or max_completion_tokens with Chat Completions. Those caps cover reasoning, visible output, and formatting tokens. They aren’t a promise about the number of visible words. Avoid treating max_tokens as an interchangeable reasoning-model control.

Recognize incomplete output

A request can succeed at the HTTP layer and still fail to produce a complete application result. Responses can return status: "incomplete" with incomplete_details.reason: "max_output_tokens". Reasoning can consume the budget before any visible answer appears. Check that state before interpreting an empty answer as a valid negative result.

Chat Completions exposes finish_reason. stop indicates complete model output, length indicates the output cap or token limit interrupted generation, and content_filter indicates omitted content. A tool_calls outcome asks the application to continue the tool loop; see 0.4. None of these fields replaces validation of the business result.

Measure time and capacity separately

Azure OpenAI documents time to first token, time between generated tokens, and time to last token. Longer generation commonly increases total latency. Streaming sends available chunks sooner and improves perceived responsiveness, but the overall generation still has to finish.

Use Azure Monitor AzureOpenAITimeToResponse for first-response responsiveness, AzureOpenAITTLTInMS for total response time, and AzureOpenAINormalizedTBTInMS for time between tokens. Pair them with GeneratedTokens and ProcessedPromptTokens. An increase in total time accompanied by a much longer answer isn’t the same evidence as slower generation at the same output length.

Tokens per minute (TPM) and requests per minute (RPM) concern workload capacity. Microsoft’s performance guidance states that standard-deployment quota governs admission and doesn’t directly enforce throughput. A request can fit its context window while the workload exceeds its admission allowance. If time to last token rises without an increase in generated-token count, check for capacity pressure. On pay-as-you-go deployments, chart Azure OpenAI Requests (AzureOpenAIRequests) for HTTP 429 throttling and concurrent request volume. On PTU-managed deployments, chart Provisioned-managed Utilization V2 (AzureOpenAIProvisionedManagedUtilizationV2). Use those signals to distinguish longer generation from deployment pressure before choosing a remedy.

Key distinctions and trade-offs

Window versus output cap: The window limits the combined context. An output cap limits generation within that context; it cannot create more room.

Variation versus reasoning effort: Sampling affects output variation where supported. Reasoning effort affects deliberation. Neither guarantees factual accuracy.

Latency versus throughput: A single response’s duration differs from how much work the deployment admits and completes across many requests.

More context can supply needed evidence but adds input consumption. More output space can prevent truncation but permits greater generation cost. More reasoning can improve difficult decisions but spend tokens without improving a routine classification. Evaluate each change against accepted results and measured usage.

Architecture / decision scenario

A document assistant becomes slower after teams attach long histories and request elaborate explanations. The architect records prompt tokens, completion tokens, reasoning-token detail, response status, and the three latency metrics for representative requests.

For a GPT-5.5 Responses request close to its documented 922,000-token effective budget, the system selects the relevant document passages and reduces redundant history before requesting generation. It gives reasoning and the final answer room to fit. An incomplete response enters an explicit application failure path rather than being stored as a finished report.

For routine summaries, the team evaluates shorter output instructions and a smaller supported generation cap. For difficult comparisons, it evaluates a supported reasoning_effort value against answer quality and cost. It changes one sampling control at a time only on compatible models. Streaming improves first-response feedback, while deployment admission and request volume remain separate capacity measurements.

Common exam traps

• Counting tokens as words or counting only the user message.
• Adding maximum input and maximum output figures as guaranteed capacity.
• Assuming a successful HTTP response implies complete generation.
• Counting visible text alone when reasoning tokens are billed as output.
• Copying sampling parameters across models without checking support.
• Treating streaming or a TPM allowance as a guarantee of low total latency.

What to remember

Budget the complete prompt, reserve generation space, and use model-specific controls. Inspect completion state as well as usage. Relate latency to generated tokens, and separate per-request context limits from deployment throughput. Optimize cost per accepted result rather than token count in isolation.

Knowledge check

References

• Microsoft, Foundry Models sold directly by Azure
• Microsoft, Azure OpenAI reasoning models
• Microsoft, Use the Chat Completions API
• Microsoft, Prompt engineering techniques
• Microsoft, Azure OpenAI performance and latency
• Microsoft, Function calling