Route inference by work and cache reuse

Learning objective

Configure load balancing for LLM serving: round-robin versus queue- and KV-aware routing.

Core idea

Equal request counts do not imply equal inference work. One request can carry a long uncached prompt while another reuses a prefix already held by a worker. A routing policy needs to decide whether simplicity, live load, or reusable key-value (KV) cache should determine placement.

NVIDIA Dynamo exposes this choice through the Frontend’s --router-mode. Its KV-aware policy combines cache overlap with projected active work. Queue admission is a separate decision: a request can wait for capacity before the router selects its worker. Confusing admission with selection produces a system that distributes overload evenly while still missing its latency target.

How it works

Establish a request-count baseline

A Kubernetes ClusterIP Service is not automatically a round-robin inference router. In kube-proxy’s iptables and nftables modes, the rules select backend Pods at random by default, without token-work or KV-cache scoring. The ipvs proxy mode offers rr and lc schedulers, but is deprecated as of Kubernetes v1.35.

python -m dynamo.frontend --router-mode round-robin cycles requests through workers. Use it as a baseline when requests have similar lengths and little reusable prefix locality. Its attraction is a placement decision that does not depend on cache-state observations.

For varied work, Dynamo also offers --router-mode least-loaded, which selects the worker with the fewest active connections, and --router-mode power-of-two, which samples two workers and selects the less-loaded one. These are load signals, not measurements of the incoming request’s uncached prompt. A worker with one very large request can have fewer active connections than a worker serving several small requests.

Keep the objective’s phrase “queue-aware” precise. Connection count is a load proxy; queued prompt tokens describe outstanding prompt work; a queue threshold controls when dispatch occurs. Those mechanisms answer different questions. Choose the signal that explains the observed imbalance rather than renaming every non-round-robin policy queue-aware.

Combine prefix locality with active load

Enable Dynamo’s cache-aware selection with --router-mode kv. Workers publish KV lifecycle events that let the router track per-worker prefix overlap. When blocks are created or removed, the corresponding observations change the cache credit used for future placement.

Dynamo’s routing concepts describe three cost terms. Prefill cost includes active prompt work plus the incoming request’s uncached work. Decode cost includes active KV blocks plus projected blocks for the request. An optional active-request cost charges a block-equivalent amount for requests already active on the worker. The router normally chooses the lowest-cost eligible worker.

This combination matters because the worker with the largest prefix hit can also have the longest backlog. Cache reuse saves redundant prefill computation, but that saving must outweigh waiting behind other work. The router’s cache credit reduces projected prefill cost; it does not make a busy worker free. The optional --router-decode-active-request-weight adds request-count pressure where decode latency is sensitive to batch size; it can also sacrifice cache locality if over-weighted.

Without worker KV events, kv mode uses load-only scoring. The alternative --no-router-kv-events predicts cache state from routing decisions. These are different operating assumptions: observed cache lifecycle and predicted cache residence are not interchangeable. Verify the chosen observation path when a KV-aware deployment behaves like a load-only balancer.

Decide whether to wait or exclude a worker

Dynamo’s --router-queue-threshold enables admission backpressure. When all eligible workers exceed the configured fraction of max_num_batched_tokens in active prefill tokens, the request waits in the router queue. The router scores it after capacity becomes available, using fresh load and cache information.

Busy thresholds have another role. --active-decode-blocks-threshold, --active-prefill-tokens-threshold, and --active-prefill-tokens-threshold-frac mark overloaded workers and exclude them from normal selection. Waiting delays a decision; busy filtering removes a candidate. If every eligible worker is overloaded, the router reports AllEligibleWorkersOverloaded. Increasing the replica pool or reducing admitted work addresses saturated capacity; switching between equivalent selection policies does not manufacture it.

A request waits while all eligible workers exceed the queue threshold, and selection then excludes overloaded workers.

Measure the correct queue. dynamo_frontend_router_queue_pending_requests records requests waiting in the router scheduler queue when queueing is enabled. dynamo_frontend_router_queue_pending_isl_tokens records their raw input tokens. A rising token backlog with a modest request count exposes a long-prompt surge that a request-count-only view can miss. Pair it with dynamo_frontend_time_to_first_token_seconds to check the effect on time to first token (TTFT).

Place the selection decision once

Dynamo supports a Frontend-embedded router, a Gateway API Endpoint Picker Plugin (EPP), and a standalone router service. Choose the topology that owns endpoint selection in your serving path. Behind an EPP that has already selected a worker, set the Frontend’s --router-mode direct; NVIDIA warns against configuring the Frontend to make another selection.

When scaling router replicas, decide whether to enable --router-replica-sync. Without it, each replica keeps an isolated active-block view, which can produce suboptimal balancing. With it, replicas exchange active-sequence lifecycle events, but synchronization is fire-and-forget and replicas can temporarily disagree. Adding router replicas does not automatically give each one a complete active-load view.

Scope the routing modes to the enabled features. With DYN_LORA_ENABLED, the routing guide supports KV, random, and round-robin modes. Direct, power-of-two, least-loaded, and device-aware-weighted modes are not LoRA-aware and fail startup. Session affinity with LoRA is supported only in KV mode. A policy that is useful for a plain worker pool is therefore not automatically valid for a LoRA deployment.

Key distinctions and trade-offs

Compare the policies against the source of uneven work:

These choices are composable: queueing can precede KV-aware selection. A queue can protect workers while making latency worse for callers; a cache-aware score can reduce computation while still overloading an undersized fleet. Evaluate latency and backlog together, then use 4.4.3 for capacity changes.

Architecture / decision scenario

An insurance research assistant serves two validated model replicas. Scenario measurements show that 80% of requests reuse a 6,000-token policy prefix, while the remaining requests carry unrelated 12,000-token documents. The TTFT target is 1 second. Under round-robin, p95 TTFT is 1.7 seconds; a matched KV-aware trial reaches 0.8 seconds at the same offered load and GPU allocation.

The serving owner selects --router-mode kv with worker KV events. A proposal to use least-loaded alone is tempting, but the trace shows that low connection counts still coincide with expensive uncached documents. Prefix overlap and prompt work explain the difference better than connection count.

During a document-import burst, both workers accumulate prompt work. The owner enables queue admission using --router-queue-threshold 0.8 (a scenario choice, not a default), then watches pending input tokens and TTFT. If the queue drives TTFT above 1 second, the result is a capacity or admission problem for this workload, even if cache-aware routing still beats round-robin. The team does not report a routing success while callers wait beyond the target.

Common exam traps

• Equating equal request distribution with equal token work.
• Assuming few active connections means little remaining inference work.
• Calling kv mode cache-aware without checking its KV observation path.
• Always sending to the worker with the largest cached prefix, regardless of load.
• Treating queue admission and overloaded-worker filtering as the same control.
• Selecting an endpoint twice or ignoring LoRA routing restrictions.

What to remember

Use round-robin as a simple baseline, load-aware selection for active-work imbalance, and KV-aware routing when reusable prefixes change placement economics. Queue admission decides when to dispatch; worker selection decides where. Validate both with token backlog and latency under the same offered workload.

Knowledge check

References

• Kubernetes, Virtual IPs and Service Proxies

• NVIDIA, Router Operations

• NVIDIA, Configuration and Tuning

• NVIDIA, Router Guide

• NVIDIA, Routing Concepts

• NVIDIA, Router Filtering

• NVIDIA, Metrics Catalog