Enforce budgets before admitting more work

Learning objective

Enforce and monitor runtime token and execution budgets using context caps, turn/tool-call counters, alert thresholds, and result trimming; distinguish enforcement from loop design.

Core idea

A runtime budget is an admission decision: can this request start another model call, tool invocation, or delegated task with the resources that remain? For architects operating multi-agent systems, the goal is to enforce that decision even when an agent keeps proposing useful work. A prompt that asks the agent to stop is an instruction, not an enforced resource limit.

Keep three quantities separate: the context sent in one inference request, cumulative consumption across the business request, and execution counts or elapsed time. A short context can still support hundreds of expensive calls. A low turn count can still trigger many parallel tools. Each quantity needs its own counter and stop condition.

See 1.1.5 for planning-loop termination and 2.2.3–2.2.4 for evidence budgets and compaction. Here the question is how to prevent another operation after the request exhausts its allowance, and how to prove that prevention happened.

How it works

Put the check inside the execution path

Use Agent Framework’s built-in limits first for each automatic tool-invocation loop. In Python, configure client.function_invocation_configuration with max_iterations for model round trips, max_function_calls for total function calls, and max_duration_seconds for elapsed wall-clock time. The latter two default to None, meaning unlimited; set positive values when the request needs those limits.

When the client reaches a limit, it stops invoking tools and asks the model for a final text response. These limits are best effort: checks happen after each model-requested batch of tools, so one batch can exceed the call-count or duration limit. Time waiting for tool approval counts toward max_duration_seconds. A final-response request also needs remaining allowance in your overall budget.

For .NET’s HarnessAgent, HarnessAgentOptions.MaximumIterationsPerRequest overrides the FunctionInvokingChatClient function-invocation limit. Its default null retains that client’s default. Do not infer Python’s call-count and duration settings from this iteration setting.

Python chat middleware checks each model request through ChatContext, while function middleware checks individual tools through FunctionInvocationContext; see 2.4.15 for the middleware layers and termination semantics. Use these admission points for the context cap and strict shared allowance that the per-client, best-effort settings cannot guarantee. Define whether each application counter measures user turns, model requests, or executed tools.

On shared-budget exhaustion, use Python’s MiddlewareTermination with the application’s terminal result. Define an outcome such as budget_exhausted instead of fabricating business success; that name is your contract, not a built-in enum. See 2.4.15 for .NET’s FunctionInvocationContext.Terminate counterpart.

Store cumulative counters in a request-owned ledger that every participating agent consults. Middleware metadata can pass information within the middleware path, but a dictionary by itself does not coordinate independent processes. For concurrent branches, reserve the next operation’s allowance atomically before dispatch. Reconcile the reservation when measured usage arrives. These ledger and reservation rules are application design, implemented around the middleware checks.

Cap the context that reaches the model

At each model-call admission, measure the assembled input, not only the user’s message. Include trusted instructions, selected history, retrieved evidence, tool schemas, and tool observations. Reserve space for the response according to the selected model’s documented interface and limits. Reject or reduce the request before dispatch if the assembled context does not fit the application cap.

Agent Framework’s TokenizerProtocol lets you supply a model-specific tokenizer for token-aware compaction. CharacterEstimatorTokenizer uses a four-characters-per-token heuristic. Treat that value as an estimate and leave headroom; it is not an exact token count for every model or language. The compaction framework is experimental in Python and .NET; .NET documents warning MAAI001.

Compaction changes what the model receives. Enforcement decides whether that changed input is admissible. Recheck the final assembled context after compaction or retrieval trimming, because a strategy’s trigger does not prove the entire request fits. Preserve tool-call/result relationships and critical business facts as covered in 2.2.4.

Trim results before they consume more context

A tool can return a large successful result that makes the next model request unaffordable. Python function middleware can modify FunctionInvocationContext.result after call_next; use that point for a tool-specific result projection. Preserve identifiers, evidence provenance, relevant fields, and an explicit indication that material was omitted. Return a reference to a separately stored full result when the application supports it.

Result trimming limits future context consumption. It does not refund a tool operation, network transfer, or inference that already ran. Likewise, trimming the final user-visible answer does not reduce tokens already generated. Apply the projection before the result enters another model call, and record the original and admitted sizes in your application telemetry.

Each model request and each tool passes a middleware check, and either check can end the loop with the application’s terminal result.

Handle streaming separately. The middleware documentation distinguishes non-streaming AgentResponse from a streaming async generator of AgentResponseUpdate. A transformation written for a completed response is not automatically valid for incremental output. Decide which fields can be emitted early and which need complete validation before release.

Observe the budget without delegating enforcement to alerts

Agent Framework emits the gen_ai.client.token.usage histogram for chat operations. Its gen_ai.client.operation.duration measures model-operation duration, while agent_framework.function.invocation.duration measures function duration. These signals explain consumption; they do not stop execution. Emit application-owned counters for admitted model calls, tool calls, budget refusals, and elapsed request time alongside them.

Use an Azure Monitor log search alert with a static Threshold value, an Operator, and an action group to notify operations before exhaustion becomes widespread; see 3.3.4 for the alert lifecycle. Choose warning thresholds below the enforced cap. Alerts are a delayed operational response, so the synchronous middleware check remains the authority on whether another operation starts.

Key distinctions and trade-offs

Loop design versus budget enforcement: A well-designed loop decides what constitutes completion or lack of progress. An enforced budget stops further work regardless of the model’s proposed plan. Both are necessary, and neither substitutes for the other.

Per-call cap versus cumulative allowance: A context cap protects one model invocation. A cumulative ledger protects the entire request, including delegated branches and repeated attempts. Resetting a counter at every handoff defeats the latter.

Warning versus refusal: A threshold alert invites investigation. MiddlewareTermination prevents the next normal execution path. A low alert rate does not prove the refusal path works.

Hard caps can interrupt a task near completion. Reserve enough allowance to format a truthful terminal response, and expose partial results only under the business contract. Larger caps reduce interruptions but increase the amount a defective loop can spend.

Architecture / decision scenario

A claims workflow calls a planner, two evidence agents, and a verifier. Each Python client sets max_iterations=8, max_function_calls=12, and max_duration_seconds=90 for its tool loop. The workflow also enforces shared application limits of eight total model requests, twelve executed tools, a 90-second deadline, and a 12,000-token input cap per request. These are policy choices, not Microsoft service limits. A warning fires at 75% of the cumulative token allowance.

Chat middleware checks assembled context and reserves a model-call slot from the shared request ledger. Function middleware reserves tool slots and projects oversized search results into evidence records. Parallel agents receive the same request budget identity; neither creates a fresh allowance on handoff. The shared checks include prior agents’ work and reserve allowance before individual tools start, closing the built-in settings’ per-client and per-batch gaps. The deadline includes earlier work and approval waits.

The verifier asks for a ninth model call. The runtime refuses it through MiddlewareTermination and returns the application’s incomplete outcome. Token and duration telemetry show where the allowance went, and the Azure Monitor alert brings the pattern to operations. The team then changes the verifier’s repeated-check behavior under 1.1.5; raising the cap alone would only permit more repetitions.

Common exam traps

• Treating a stop instruction as an enforced execution limit.
• Counting only agent invocations when model calls repeat inside them.
• Treating max_function_calls as a hard cap that a parallel batch cannot exceed.
• Resetting cumulative allowances on delegation or retry.
• Treating estimated context size as an exact model token count.
• Trimming displayed output after generation and expecting a cost refund.
• Waiting for an alert to prevent the next synchronous operation.

What to remember

Configure built-in tool-loop limits, then check strict admission at the model and tool execution points. Enforce per-call context caps and request-wide counters separately, reserve allowance across concurrent branches, and trim observations before the next inference. Use telemetry and warning thresholds to explain and improve the enforced policy; use loop design to eliminate needless work.

Knowledge check

References

• Microsoft, Using function tools with an agent
• Microsoft, Agent Middleware
• Microsoft, Compaction
• Microsoft, Observability
• Microsoft, Create Azure Monitor log search alert rules