Budget one agent turn

Learning objective

Reason about context, latency, and cost within the budget of one agent turn. This lesson covers turn-level resource planning, not model selection or end-to-end production optimization.

Core idea

Treat an agent turn as a bounded execution envelope. The turn starts with a user request and ends with a response, an explicit pause, or a failure. Everything the agent reads, generates, retrieves, and invokes during that interval consumes one or more budgets:

• Context budget: the information the model can process in one model call.
• Latency budget: the time available before the result becomes too slow.
• Cost budget: the permitted spend on model input, output, retrieval, tools, and infrastructure.

These budgets interact. Adding retrieved documents can improve evidence but uses context, increases model processing, and may add retrieval latency. Calling a faster model can reduce latency and cost but may reduce reasoning quality. An architect therefore allocates budgets before execution and records actual use afterward.

How it works

First, distinguish an agent turn from a model call. One turn may contain several model calls and tool calls. For example, a model may choose a search tool, receive its result, and call the model again to compose the answer. The turn budget covers the whole sequence, not only the final generation.

For each model call, the context includes more than the latest user message. It can include system instructions, relevant conversation history, tool definitions, retrieved evidence, tool results, and output generated so far. Vertex AI’s countTokens method accepts content, tools, and system instructions, which reflects that all of these items contribute to the request sent to the model.

A practical turn budget has four steps:

1. Reserve fixed context. Account for system instructions, tool schemas, and mandatory policy text.
2. Allocate variable context. Divide the remaining space among history, retrieved evidence, tool results, and the expected answer. Keep a safety margin for unexpected results.
3. Allocate time. Set deadlines for model calls, retrieval, tools, and retries. A parallel group consumes roughly the slowest branch’s time, while a sequential chain accumulates the latency of its steps.
4. Allocate spend. Estimate the number of calls and the input and output units each call processes. Include paid tools and supporting services. Check the active model’s official pricing because units and rates differ by model and can change; see Vertex AI generative AI pricing.

Enforce the limits in code. Use call deadlines, maximum tool invocations, maximum iterations, response-length limits, and explicit stop states. When the remaining budget cannot support another useful step, the agent should answer from sufficient evidence, ask for necessary information, or stop with a bounded failure. It should not start an action that it cannot finish safely.

Key distinctions and trade-offs

Context window versus usable context: A model’s context window is a technical limit. Usable context is smaller because the architecture must reserve space for instructions, tool exchanges, and output. Filling the window is not a goal; irrelevant content can make the important evidence harder to use.

Latency versus throughput: Latency measures how long one request takes. Throughput measures how much work the system completes over time. Batching or high concurrency can improve throughput while making an individual request wait longer.

Maximum cost versus expected cost: A typical path may use one model call, but a retry or tool loop can use several. Expected cost supports capacity planning; the maximum enforced cost limits worst-case exposure.

Useful optimizations also have conditions. Parallelize only independent calls. Cache only content whose reuse, freshness, and access rules are understood. Summarize history only when the loss of detail is acceptable. Use retrieval to select evidence rather than placing an entire corpus in every prompt.

Architecture / decision scenario

A support agent has an eight-second service-level objective. It needs policy evidence and may query an order system. The architect assigns two seconds to retrieval, two seconds to the order lookup, and four seconds to model processing and response delivery. Because retrieval and order lookup are independent after intent detection, the system runs them in parallel and gives the group a two-second deadline.

The prompt reserves fixed space for safety instructions and tool schemas, then limits history and retrieved passages. The controller permits one tool round and one retry only for a transient tool failure. If the order service misses its deadline, the agent explains that live order data is unavailable rather than entering another reasoning loop.

This design makes degradation explicit. A slow dependency reduces answer completeness, but it does not create unbounded latency or spend.

Common exam traps

• Treating the context window as space only for user text. Instructions, history, tool schemas, results, evidence, and generated tokens also need capacity.
• Estimating a turn from one model call when the agent can call tools and the model repeatedly.
• Assuming parallel work has no latency. Fan-out is bounded by the slowest branch and still needs a join deadline.
• Optimizing token count while ignoring retrieval, tool, and infrastructure cost.
• Relying on a prompt such as “be brief” as a hard budget. Enforce hard limits in the controller and API configuration.
• Retrying without a cap or retrying permanent failures as if they were transient.

What to remember

• Budget the whole turn, including every model, retrieval, and tool call.
• Reserve context for fixed instructions and output before adding variable data.
• Sequential latency accumulates; independent parallel work waits for its slowest required branch.
• Track expected use for planning and enforce maximum use for safety.
• Define a useful degraded result and a stop condition before the turn starts.

Knowledge check

References

• Google Cloud, Count tokens for Gemini models
• Google Cloud, models.countTokens REST method
• Google Cloud, Generative AI on Vertex AI pricing
• Google Cloud Architecture Center, Optimize AI and ML workloads