Allocate tokens before admitting each request

Learning objective

Budget the context window across system prompt, tool definitions, retrieved content, history, and output, and forecast usage with token counting.

Core idea

For an architect operating a Claude service, a context budget is an admission decision: which request can complete with enough room for its answer? Allocate capacity across the assembled input and output before inference, rather than treating the model’s window as an input allowance.

Everything supplied in system, tools, and messages consumes context, including retrieved documents and tool results. Generated output consumes the same window. A request can fit at admission yet run out of capacity while generating. Your request builder owns the allocation; the service owner accepts its effects on task completion, latency, and cost.

How it works

Allocate one shared window

Choose the deployed model and verify its context and output limits. For example, Anthropic documents a 1M-token context window and a maximum of 128k output tokens for Claude Sonnet 5.5. Those limits establish available capacity, not a target request size. An application’s smaller operating budget can reflect its latency or spend constraints.

Represent the budget by component so growth has an identifiable owner:

These are accounting categories you define, not extra API fields. Documents and history both enter messages; label them in your builder without counting the same content twice. Tool results belong to history after the application appends them. Their cost can recur on subsequent turns.

Use the planning rule estimated input + output allowance + safety margin <= context capacity. The margin is an application choice that covers estimation differences and foreseeable input growth. max_tokens caps generated output; it does not guarantee that the whole allowance remains available when a request approaches the window limit. Thinking tokens are part of that cap, so an answer-only output allowance understates demand when thinking is enabled. See 2.1.3 for choosing reasoning settings.

Count the request you actually plan to send

The Claude API operation is POST /v1/messages/count_tokens. In Python, client.messages.count_tokens(...) accepts the target model and the structured inputs, including system, client tools, and messages. Its response’s input_tokens estimates total input demand. Use the same assembled payload and target model as inference; a word count or a forecast from another model is not equivalent.

Token counting is free, with rate limits independent of message creation. It remains an estimate: Anthropic says the actual count can differ slightly and can include system-added tokens that are not billed. Treat the endpoint as a preflight measurement, then reconcile it with response usage. Recount after changing instructions, attached tools, selected documents, or the history representation.

The endpoint has narrower input support than Messages. It accepts client tools and the advisor tool, but rejects other server tools and the MCP connector. Images and documents with url or file sources also produce invalid_request_error; supply base64 images or PDFs for counting. For integrations with unsupported inputs, use actual Messages usage to calibrate an application forecast rather than claiming the endpoint counted the complete request. An advisor-tool count covers only the executor’s first sampling call.

Reconcile capacity with observed consumption

For a response using prompt caching, input is split across usage.input_tokens, usage.cache_read_input_tokens, and usage.cache_creation_input_tokens. Add all three when measuring window occupancy. A cache hit reduces the price of repeated input; those tokens still occupy context. Cache economics belongs to 2.5, while this budget counts capacity regardless of billing category.

Add the generated usage.output_tokens when forecasting the following turn, because the previous response becomes part of its history. Prior thinking blocks also remain input by default on Claude Opus 4.5 and later Opus models, Sonnet 4.6 and later Sonnet models, and Haiku 5.5; earlier Opus/Sonnet models and Haiku through 4.5 strip thinking from earlier completed turns. Include any new tool result and user content before admitting that turn. To override thinking retention, configure clear_thinking_20251015 in context_management.edits with the beta header context-management-2025-06-27. A budget checked only on the first request misses the accumulation that creates long-run failures.

Observe forecast error by model and input shape, component growth, and the terminal outcome. If input alone exceeds the window, every model returns HTTP 400 invalid_request_error with “prompt is too long.” On Claude 4.5 and later models, input plus max_tokens can exceed the window without immediate rejection; generation that reaches the boundary ends with stop_reason: "model_context_window_exceeded". Reduce or compact input before retrying a capacity failure, or use an evaluated larger-window path. Repeating the unchanged request does not create space. Distinguish that full-window outcome from stop_reason: "max_tokens", which means generation reached the per-request output cap; if the result is incomplete, raise max_tokens or continue the response within the remaining window.

Give the model a pacing target

Your application admits requests and enforces the run’s cost and latency limits; model-visible budgets help Claude pace its work. Claude Sonnet 5, Sonnet 4.6, Sonnet 4.5 (deprecated), and Haiku 4.5 receive automatic <budget:token_budget> and per-tool-call <system_warning> context-awareness tags; Sonnet 5.5, Opus 4.7 and later Opus models, and Haiku 5.5 do not receive those tags.

On supported Messages API models such as Sonnet 5.5, beta header task-budgets-2026-03-13 enables output_config.task_budget: {"type": "tokens", "total": N}, with a minimum total of 20,000 tokens. This is an advisory target across the agentic loop, not an enforced spend cap; max_tokens remains the hard per-request output limit. Task budgets are not supported on Sonnet 5, Sonnet 4.6, Opus 4.6, or Haiku 4.5, or on Claude Code and Cowork surfaces.

Key distinctions and trade-offs

Window limit versus operating budget: The vendor limit defines what is possible. The application budget defines what the service can afford and finish reliably.

Count versus reserve: input_tokens forecasts input; max_tokens limits output. Neither establishes answer completeness by itself.

Component estimate versus complete count: Counting variants with and without tools helps attribute growth, but the assembled request’s count is the admission measurement. Formatting and service-added tokens mean handwritten component sums need reconciliation.

Tight allocations reduce repeated input and make performance more predictable, but can exclude required evidence or truncate useful output. Increase the component whose shortage causes measured failures, rather than raising every allowance. See 2.4.2 for selecting what enters and 2.4.3 for managing accumulated history.

Architecture / decision scenario

A financial-services analyst assistant must return a source-backed comparison within a 12-second application target. Its team uses Claude Sonnet 5.5, but replay measurements show that a 140,000-token input takes 21 seconds at the 95th percentile, while a curated 55,000-token input takes 9 seconds with the required comparisons intact. These are illustrative workload measurements, not vendor latency promises.

A proposed fix is to accept everything that fits the 1M window. The service owner rejects it because capacity does not satisfy the latency target. The architect allocates 5,000 tokens to system, 4,000 to tools, 28,000 to retrieved excerpts, and 18,000 to history, with max_tokens: 6000 and a 3,000-token safety margin. The 64,000-token planning total is well below capacity, while the 55,000-token input target reflects measured service performance.

The builder counts the complete request against the deployed model, curates material if input exceeds the operating target, and rechecks after each tool cycle. Operations compares forecasts with all three input usage fields and output usage. If a server tool is introduced, its unsupported counting path becomes a measured forecast rather than a false preflight guarantee. The owner accepts budget changes only after task coverage and end-to-end latency remain within target.

Common exam traps

Avoid these budget mistakes:

• Spending the entire window on input and forgetting generated output.
• Assuming a cache hit removes tokens from context.
• Counting words or using another model’s tokenizer.
• Treating token counting as support for every Messages input.
• Assuming acceptance of input plus max_tokens guarantees a complete answer.
• Checking only the first turn of a growing agent run.
• Treating an advisory task budget as an enforced run-spend cap.

What to remember

Allocate input and output together, count the assembled request against its target model, and reconcile the forecast with all input usage categories. Keep the application’s latency and quality budget distinct from the model’s maximum capacity.

Knowledge check

References

• Anthropic, Context windows
• Anthropic, Token counting
• Anthropic, Context editing
• Anthropic, Task budgets
• Anthropic, Stop reasons and fallback