Skip to lesson

Google CloudProfessional Agentic Architect2026-094.2 Deploying and scaling production workloads

Size an agent runtime for production load

Completed by the knowledge check

Learning objective

Size a chosen agent runtime for concurrency, capacity, scaling behavior, and startup latency. Runtime selection and model-endpoint sizing are out of scope.

Core idea

Size a runtime as a capacity envelope, not as a CPU and memory setting. The envelope has four connected values:

  • Safe concurrency per instance: requests one warm instance can handle while meeting latency and error objectives.
  • Warm floor: the capacity that is ready before demand rises.
  • Scale ceiling: the largest instance or Pod count the runtime may create.
  • Startup time: the delay before added capacity can accept work.

Autoscaling operates inside this envelope; capacity is neither unlimited nor instantaneous. A high maximum cannot prevent startup latency.

How it works

Start with concurrent in-flight work, not only requests per second. If requests arrive at rate R and each occupies a runtime slot for T seconds on average, their average concurrency is approximately:

in-flight requests = R * T

Model, retrieval, and tool waits increase T despite low CPU use. Asynchronous code can overlap these turns, but each consumes memory, connections, and quota.

Determine the envelope in this order:

  1. Measure one-instance capacity. Load test a warm instance with representative turns. Increase concurrency until latency, errors, CPU, memory, connections, or a downstream limit approaches its threshold. Choose a safe value below that breakpoint. Google recommends this preparation before large-scale Cloud Run load tests.
  2. Estimate steady-state instances. Divide in-flight work by safe per-instance concurrency, round up, and add headroom. Validate with a sustained test; averages hide bursts and slow-tail requests.
  3. Set the warm floor from the latency objective. Keep enough instances or Pods ready to absorb normal traffic and any burst that cannot wait through startup. Pre-scale for predictable events. A queue and backpressure can trade response time for a lower warm floor. Google Cloud recommends planning for both known peaks and unexpected demand.
  4. Set the scale ceiling from end-to-end limits. Confirm runtime quotas, regional capacity, model and tool quotas, database connections, and cost. The smallest constraint is the usable ceiling. Protect dependencies with admission control or load shedding.
  5. Test the scaling transition. Run warm, cold-start, spike, ramp, and soak tests. Observe queue and startup time, instance count, saturation, errors, and latency percentiles. Capacity is load served within the service objective, not load accepted before failure.

Key distinctions and trade-offs

Concurrency versus parallelism: Concurrency counts overlapping requests; parallelism counts work executing at the same instant. An asynchronous, I/O-bound agent can support high concurrency with limited CPU parallelism. A CPU-bound or non-thread-safe component can require lower concurrency.

Configured concurrency versus safe concurrency: A platform concurrency setting is a routing limit, not proof that the application can sustain it. Cloud Run recommends lower values when each request consumes most CPU or memory. Lower concurrency reduces contention but needs more instances; higher concurrency can improve utilization but increase in-instance queuing. See Cloud Run maximum concurrency.

Warm capacity versus elastic capacity: Minimum instances or replicas reduce exposure to cold starts but reserve more capacity. Maximum instances bound cost and dependency pressure, but requests can queue or fail at the ceiling.

Scaling signal versus resource bottleneck: CPU is a poor scaling signal for an agent that spends most of its time awaiting models and tools. Request concurrency, queue depth, or capacity metrics can react more directly. Cloud Run scales from CPU and request concurrency signals; configured concurrency remains an upper bound, not promised throughput. See Cloud Run instance autoscaling.

The chosen runtime determines where these controls apply:

RuntimePer-instance or Pod controlsScaling path to account for
Agent RuntimeCPU, memory, and container concurrencyManaged scaling between minimum and maximum instances; higher concurrency can use asynchronous workers but can exhaust memory.
Cloud RunCPU, memory, and maximum requests per instanceRequest and CPU signals add instances between minimum and maximum settings. Scale-from-zero and burst scale-out can add startup and pending time.
GKEContainer requests and limits, application worker count, and Pod concurrencyHorizontal Pod Autoscaler (HPA) adds Pods; if existing nodes cannot place them, cluster autoscaling must add nodes. GKE cluster autoscaling uses Pod resource requests, not actual utilization. See GKE cluster autoscaling.

For GKE, size both layers. An HPA ceiling does not guarantee schedulable Pods, and a node-pool ceiling does not configure replicas. Choose an HPA metric that reflects the bottleneck; GKE supports resource, custom, and external metrics. See GKE horizontal Pod autoscaling.

Architecture / decision scenario

A support agent receives 24 requests per second during a predictable morning peak. A representative turn occupies the runtime for an average of five seconds, so expected in-flight work is 24 * 5 = 120 requests. A one-instance test shows that six concurrent turns meet the latency objective; at eight, memory pressure and tail latency rise sharply.

The steady-state estimate is 120 / 6 = 20 warm instances. The team provisions 25 for the known peak to provide 25% headroom. It sets a higher ceiling only after checking model and database quotas.

A spike test shows that new instances need eight seconds to become useful. Ten warm instances would provide only 60 immediate slots, so autoscaling from ten cannot meet the strict latency objective. The team keeps 25 warm before the scheduled surge. It applies admission control to unexpected excess traffic and queues only operations that can wait.

On GKE, the same calculation sets the warm Pod count, but nodes must have room for those Pods. Otherwise, scale-out includes Pod creation, node provisioning, image startup, and readiness before capacity arrives. GKE exposes startup latency metrics for these stages.

Common exam traps

  • Treating requests per second as capacity without accounting for request duration and in-flight work.
  • Setting platform concurrency to its highest value and assuming that value is safe application capacity.
  • Assuming maximum instances are warm or autoscaling precedes a burst.
  • Scaling an I/O-bound agent only from CPU while queueing and latency increase.
  • Sizing runtime capacity without checking model, database, tool, regional, and quota ceilings.
  • Configuring GKE HPA replicas without ensuring that cluster nodes can schedule the new Pods.

What to remember

  • Convert request rate and runtime duration into concurrent in-flight work.
  • Measure safe concurrency at the required latency and error objectives.
  • Size the warm floor for demand that cannot wait through startup.
  • Treat autoscaling as a delayed control loop bounded by a scale ceiling.
  • Validate the entire dependency chain with realistic load and spike tests.

Knowledge check

References