AWSAWS Certified Generative AI Developer - Professional2026-102.2 Implement model deployment strategies
Escalate only when routine inference fails
Completed by the knowledge check
Learning objective
Apply model cascading to handle routine queries without unnecessary large-model inference
Core idea
For an architect reducing inference expense, a model cascade spends expensive capacity only where the task requires it. A smaller model handles routine requests. A controlled decision sends harder or unacceptable cases to a stronger model. The saving comes from avoiding large-model calls, not from adding a small-model call in front of every large-model call.
There are two distinct designs. A sequential cascade invokes the smaller model, checks its output, and escalates if that output fails the acceptance contract. Predictive routing selects a model before generating an answer. Amazon Bedrock intelligent prompt routing implements the second design for supported models within a model family. It predicts response quality from the prompt and forwards the request to the chosen model.
Choose between them from the evidence your decision needs. If a prompt-level prediction is sufficient, predictive routing can avoid an extra generation. If the decision requires inspecting the actual response, use an application-controlled cascade and budget for the extra work on escalated requests. Neither design turns estimated quality into proof that an answer is correct.
How it works
Define the routine path and acceptance gate
Start with a workload slice the smaller model demonstrably handles: a bounded summary, an allowed classification, or an answer grounded in supplied evidence. State the required output and risk limits before selecting the escalation rate. A cheap first stage that produces fluent but unchecked answers creates a quality problem rather than an optimization.
For an application-controlled cascade, invoke the small Bedrock model through the modelId of a Converse request, or a separately hosted model through its serving interface. Keep the model-specific interface behind the validated application contract described in 1.2.3. A successful API response means inference completed; it does not mean the business answer passed.
Validate the result against explicit conditions: required fields, allowed values, supported claims, or an independent task-quality check. Give the resulting application decision a named value such as accept, escalate, or abstain. These are application-defined values, not built-in Bedrock routing outputs. A request outside the supported task can go directly to the stronger route or to a terminal outcome without spending a small-model generation.
AWS Step Functions provides a concrete mechanism for orchestrating those decisions. A Choice state evaluates rules in Choices and follows their Next destinations. Give it a Default destination for unmatched or invalid decision values. Without a matching rule or a Default, the state fails to transition. Put the acceptance decision into workflow state explicitly; do not expect Step Functions to infer quality from model text.
Do not use the same model’s uncalibrated claim of confidence as the sole gate. Validate the gate against reviewed answers, especially where a false acceptance hides a serious error. An evaluator model also consumes latency and inference budget, so include it in the comparison.
Bound escalation before adding it
Preserve the original request, authorized evidence, and validated task context when escalating. A stronger model can reassess the task; it should not be forced to accept the small model’s unsupported answer as authoritative evidence. If you include that draft, label it as a draft and apply the same final acceptance checks.
Use a finite design: at most one small-model generation and one stronger-model generation for the ordinary sequential path. This is an example application budget, not an AWS quota. Decide the terminal outcome if the stronger answer also fails: abstain, ask a useful clarification, or send the case for review. Repeatedly escalating through more models can exhaust the deadline while producing correlated mistakes.
A request makes at most one small-model and one stronger-model generation before it is accepted or reaches a terminal outcome.
Treat transport failures separately from quality failures. A timeout or throttling response requires the bounded recovery policy in 1.2.5. An unacceptable answer requires the quality route defined here. Sharing an overall deadline and attempt budget prevents retry handling from silently multiplying the cascade.
Release routing and acceptance changes as evaluated configurations. A lower threshold can reduce escalations while increasing false acceptance. A stricter threshold can improve accepted quality while removing the economic benefit. Measure both effects before promotion.
Use intelligent prompt routing where its scope fits
Bedrock intelligent prompt routing provides default and configured routers. Its documented console flow requires exactly two models within the same family. Default routers use select Amazon Nova and Meta Llama models, with availability depending on Region. Check supported model and Region combinations rather than assuming that any two Bedrock models can be paired.
CreatePromptRouter defines the router with models, fallbackModel, and routingCriteria.responseQualityDifference. The target objects use modelArn. The fallbackModel is the baseline used when the routing criterion is not met; it is not an application disaster-recovery model selected after an outage.
The responseQualityDifference field has a documented range of 0 to 100. It governs the predicted quality difference used in routing; it is not an absolute correctness threshold or a probability that the returned answer is safe. Select it through task evaluation rather than copying an example value.
Use the prompt router ARN in the inference request’s modelId when calling Converse. The response provides the selected model through trace.promptRouter.invokedModelId. Record that field with the outcome so you can compare routing decisions with reviewed quality. Converse also returns usage and metrics.latencyMs, which help measure token consumption and request latency. They do not establish business correctness.
AWS documents that intelligent prompt routing is optimized only for English prompts and cannot adjust decisions from application-specific performance data. Specialized tasks still need your own evaluation. Do not describe the managed router as a system that learns from your acceptance labels or as a sequential small-answer inspection service.
Key distinctions and trade-offs
The following mechanisms make different decisions:
| Pattern | Decision evidence | Work paid on an escalated request |
|---|---|---|
| Sequential cascade | Actual first-stage output and application validation | Small-model call, checks, and stronger-model call |
| Predictive routing | Prompt and predicted relative response quality | Routing and the selected model’s inference |
| Failure fallback | Invocation error and recovery policy | Failed or retried attempts and the recovery call |
A sequential cascade can inspect actual errors but adds latency to escalated requests. Predictive routing avoids the first answer but depends on how well its predictions fit the task. Failure fallback handles availability, not whether a routine query needs a stronger model.
For a sequential design, expected inference work includes every small-model call plus the escalation fraction multiplied by stronger-model work. Add evaluation, orchestration, and retry costs. When nearly all requests escalate, the cascade pays for both generations and can cost more than direct stronger-model inference.
Architecture / decision scenario
An English-language product assistant receives many routine summaries and fewer difficult policy questions. The team evaluates direct strong-model inference, a sequential cascade, and a supported Bedrock prompt-router pair using the same reviewed requests.
For the sequential candidate, a Step Functions Choice follows the application’s accept or escalate result; Default sends malformed decisions to a safe terminal path. The smaller model’s answer must pass the defined evidence and output checks. A stronger-model failure does not trigger another unbounded escalation.
For the predictive candidate, the team records trace.promptRouter.invokedModelId, usage, latency, and reviewed outcomes. It adjusts responseQualityDifference through offline comparison rather than interpreting it as answer confidence. Specialized policy questions bypass the routine route when evaluation shows that prompt-level routing misses their risk.
The team chooses the candidate that reduces cost per accepted answer while preserving the quality floor and tail-latency target. If the sequential gate needs an expensive evaluator and escalates most requests, direct strong-model inference wins. If the managed router’s predictions fit routine English traffic, it can save inference work without generating a small answer first.
Common exam traps
- Invoking the large model for every request after calling a small model.
- Treating a model’s self-reported confidence as calibrated acceptance evidence.
- Confusing intelligent prompt routing with sequential answer validation.
- Treating
fallbackModelas outage recovery orresponseQualityDifferenceas correctness. - Assuming cross-family or unrestricted regional router support.
- Measuring average savings while ignoring escalated-request tail latency.
What to remember
A cascade needs a trustworthy acceptance gate, a bounded escalation path, and a measured economic benefit. Bedrock intelligent prompt routing predicts which supported model to invoke; an application cascade evaluates an actual answer. Keep quality escalation separate from failure recovery, and preserve a safe terminal outcome when stronger inference cannot satisfy the contract.
Knowledge check
A sequential cascade sends each request to a small model, runs an evaluator-model check on the answer, and escalates failures to a large model. In production, 85 percent of requests escalate. How should the team decide whether to keep the cascade?
In a cascade, the small model ends each answer with “Confidence: high” or “Confidence: low”, and the team plans to accept every high-confidence answer. Many accepted answers cite policies that are not in the supplied evidence. What should gate acceptance?
Converse response from the small modelA team must decide which model answers each question. An answer is acceptable only if every claim it makes is supported by the retrieved passages, which can only be judged once an answer exists. Which design fits that requirement?
fallbackModel is the larger model of the pairresponseQualityDifference valueIn a sequential cascade, a hard policy question fails acceptance on the small model, escalates, and the large model’s answer fails acceptance as well. The ordinary path allows one generation from each model. What should the workflow do?
A team uses a Bedrock prompt router for English customer questions and stores reviewed accept or reject labels for every answer. It expects the router to route better as labels accumulate. What should the team do instead?
trace.promptRouter.invokedModelId with each label and re-tune the router offlineresponseQualityDifference to 100 so the router always chooses the stronger modelresponseQualityDifference through offline evaluation; the value is a routing criterion from 0 to 100, not a correctness threshold.