Google CloudProfessional Agentic Architect2026-093.1 Designing and building agentic workflows in code

Choose between an LLM and an SLM

Completed by the knowledge check

Learning objective

Choose a large language model (LLM) or small language model (SLM) for each agent workload based on reasoning demand, latency, throughput, and cost. This lesson focuses on model capacity, not on self-hosted versus software-as-a-service sourcing or open versus proprietary licensing.

Core idea

Choose the smallest model that clears the capability floor while staying inside the resource envelope.

The capability floor is the minimum acceptable task quality. The resource envelope sets maximum latency and cost and minimum sustainable throughput. A candidate must satisfy all four conditions:

reasoning quality >= required quality
p95 latency <= latency objective
sustained throughput >= peak demand
cost per successful task <= budget

An SLM and an LLM are capacity tiers, not precise parameter-count categories. Parameter count can identify candidates, but it does not prove that one fits a task. Google Cloud notes that a larger model in the same family can learn more complex patterns, but can also have higher latency and cost. It therefore recommends choosing the most affordable model that meets measured quality and latency requirements. See Develop a generative AI application.

How it works

Classify the reasoning demand. Bounded classification, extraction, routing, formatting, and templated summarization often favor an SLM. Ambiguous requests, multi-step planning, constraint reconciliation, novel problem solving, and recovery from tool failures often favor an LLM. Prompt length is not reasoning demand: a long document can require simple extraction, while a short request can require difficult planning.
Set a task-specific quality floor. Build an evaluation set with normal, difficult, and boundary cases. Score the behavior the agent needs, such as correct tool choice, valid structured output, or successful task completion. Google Cloud recommends diverse examples aligned with the task and a combination of metrics and human evaluation. See Evaluate a model.
Test the SLM first. Keep the evaluation cases and tool contracts stable, and record each candidate’s prompt and settings. If the SLM clears the quality floor, a larger model needs a measurable benefit to justify its extra resources. If the SLM fails important reasoning cases, test an LLM rather than hiding the gap with averages from easier cases.
Load-test qualified models. Measure latency and throughput with realistic input lengths, output lengths, concurrency, and traffic bursts. For streaming interactions, time to first token (TTFT) affects perceived responsiveness; request latency measures completion. Throughput measures total work over time. Requests per second alone can mislead because requests contain different token counts. Google Cloud defines useful inference measures such as TTFT, request latency, input tokens per second, and output tokens per second in About AI/ML model inference on GKE.
Calculate effective cost. Apply the candidate’s pricing unit to the real input and output mix, then include retries and escalation caused by failed attempts. Compare cost per successful task, not only list price per token or cost per request. A cheap model that repeatedly fails or produces excess output can cost more for the completed outcome.
Route when one tier does not fit every request. Use an SLM for request classes that meet the quality floor and send only difficult or high-risk classes to an LLM. Google Cloud’s agentic architecture guidance recommends model routing by task complexity, cost, or latency and gives the same small-model/simple-task and powerful-model/complex-reasoning pattern. See Choose your agentic AI architecture components.

Key distinctions and trade-offs

Decision signal	When an SLM tends to fit	When an LLM tends to fit
Reasoning demand	Stable intent classes, fixed schemas, narrow tools, explicit rules	Ambiguity, interacting constraints, longer plans, unfamiliar cases
Latency	A tight interactive deadline after quality is met	Extra response time is acceptable for harder reasoning
Throughput	High-volume work must fit serving capacity	Lower volume or more capacity can support heavier inference
Cost	Many low-value or repetitive requests make unit cost dominant	Better reasoning prevents expensive errors, retries, or human work

Latency and throughput are not synonyms. Latency describes how long one request takes; throughput describes how much work the service completes per unit of time. A system can increase total throughput through concurrency while queueing makes tail latency worse. Size capacity from the token mix as well as the request rate. For example, Vertex AI calculates provisioned throughput from input and output units per query multiplied by queries per second, not from query count alone. See Calculate Provisioned Throughput requirements.

An SLM is not automatically fast or cheap in every deployment, and an LLM does not guarantee a correct answer. Serving hardware, batching, prompt size, output size, reasoning settings, and retries affect the observed result. Benchmark the exact model and configuration under the expected workload.

Architecture / decision scenario

An expense agent receives receipts and employee questions. Most requests ask it to extract fields, assign one of a fixed set of expense categories, and call a single validation tool. A smaller model meets the required structured-output and tool-selection quality, stays below the interactive latency target, and sustains the peak receipt volume at low cost.

Some requests contain conflicting regional policies, missing evidence, or a multi-step exception that requires several tools. On this segment, the smaller model misses the quality floor. A larger model meets it, but using that model for every receipt breaks the cost and throughput limits.

The architect routes routine requests to the SLM and difficult policy cases to the LLM. The router uses observable signals such as request type, policy conflict, and missing required fields. Uncertain classifications go to the LLM. The team evaluates each route separately and load-tests the combined traffic mix. This design spends additional capacity only where the task requires it.

Common exam traps

Choosing the model with the largest parameter count without testing task quality.
Choosing an SLM only because its list price is lower while ignoring retries, escalation, and failed tasks.
Treating a long context as proof of high reasoning demand.
Using average latency when the requirement is a tail-latency objective such as p95.
Treating requests per second as comparable when prompt and response lengths differ.
Assuming one model tier must serve every request when routing can separate routine and complex work.
Assuming a fast single-request benchmark proves sufficient throughput at peak concurrency.

What to remember

Choose the smallest model that meets the task’s measured quality floor.
Evaluate reasoning demand from task behavior, not prompt length or model labels.
Measure tail latency and sustained throughput separately under realistic load.
Compare cost per successful task, including retries and escalation.
Route routine work to an SLM and complex work to an LLM when one tier cannot satisfy every segment.

Knowledge check

References

Google Cloud, Develop a generative AI application
Google Cloud Architecture Center, Choose your agentic AI architecture components
Google Kubernetes Engine, About AI/ML model inference on GKE
Google Cloud, Calculate Provisioned Throughput requirements