Choose routing signals behind one contract

Learning objective

Select static, content-based, or metric-driven model routing behind a model-agnostic request contract.

Core idea

Model routing selects an eligible inference target for a request. A model-agnostic contract keeps callers focused on the task they need completed, while the gateway or application owns target selection and model-specific adaptation. The contract should expose meaningful capabilities and constraints, not require every caller to understand every model’s native payload.

Choose the routing signal from the decision you need to make. Static routing provides a predictable assignment. Content-based routing uses attributes of the request. Metric-driven routing uses measured operating conditions or evaluation evidence. These policies can coexist, but eligibility must precede optimization: a faster or cheaper model is irrelevant if it cannot satisfy the request’s required capability or processing boundary.

How it works

Establish the contract before the policy

Define application fields for the task, content, response format, and allowed operating constraints. For example, a document service might accept a task class and evidence, then return a normalized result, completion status, and correlation identifier. Those are application-defined fields, not a native Amazon Bedrock schema.

An adapter translates the contract into an inference operation. Bedrock Converse offers a consistent message interface through fields such as messages, system, inferenceConfig, and modelId. It also exposes additionalModelRequestFields for model-specific parameters. A consistent API does not mean every model supports identical modalities, tool behavior, or response features. Reject an incompatible target before constructing the invocation.

Keep model selection inside the trusted service boundary. A caller can request a capability or service tier; it should not gain access to an unapproved model by supplying an arbitrary modelId. The target allowlist and adapter version remain server-owned. See 1.2.3 for releasing routing configuration behind a validated contract and 1.2.4 for processing-location boundaries.

Use static routing when predictability decides

A static policy binds a request class, tenant tier, or application release to one approved target. It does not inspect changing runtime metrics to make each selection. The application resolves the configured target and supplies its modelId to Converse.

Static routing is useful when a narrow task has one validated model, when an audit requires a reproducible assignment, or when you are establishing a baseline. It reduces decision complexity and makes quality regressions easier to attribute. Its disadvantage is that every eligible request follows the assignment even when another model could serve a routine request more economically.

Static does not mean hard-coded forever. A reviewed configuration can change the assignment between releases. Preserve the configuration revision with each request so you can explain which policy selected the model. Do not mistake a controlled configuration rollout for metric-driven per-request routing.

Use content-based routing for task differences

A content-based policy uses request attributes such as task class, language, modality, or validated complexity indicators. Simple explicit task fields often provide a more inspectable signal than an extra model classifier. When classification is necessary, validate the classifier’s output against the route allowlist and define an outcome for uncertain classifications.

Amazon Bedrock intelligent prompt routing is a managed content-aware option that predicts response quality within the same model family, is optimized for English prompts, and cannot adjust routing from application-specific performance data. See 2.2.9 for its API fields and fallbackModel semantics.

Evaluate a content-aware router against representative task slices. A classifier that sends most traffic to the cheaper target can still fail on a small, consequential request class. The acceptance decision must inspect returned quality, not only the distribution of selected models.

Use metrics when operating evidence decides

A metric-driven policy consumes measurements and updates target eligibility or preference. For Bedrock Runtime, CloudWatch publishes InvocationLatency, InvocationThrottles, InvocationClientErrors, InvocationServerErrors, InputTokenCount, and OutputTokenCount under AWS/Bedrock. The ModelId dimension lets you examine the relevant model rather than relying on an aggregate across targets. However, failures before Bedrock determines the target model appear only without that dimension, so model-filtered metrics can undercount errors. Compare them with the dimensionless series to include pre-routing failures.

Throttled requests are excluded from Invocations, InvocationClientErrors, and InvocationServerErrors. Track InvocationThrottles separately; an error-rate signal based only on those error metrics misses throttles, and its Invocations denominator excludes them too.

A successful Converse response also provides metrics.latencyMs and token usage. These are useful per-call observations, while gateway timing captures the broader user-visible path. Define which latency you are optimizing: model-call latency and end-to-end response latency are not interchangeable.

The routing policy is your application logic; CloudWatch publishes observations, not an automatic model choice. Use explicit measurement windows, sample requirements, and recovery conditions. For example, remove a target from preference only after a sustained error-rate breach, and restore it only after a separate recovery condition passes. This prevents a noisy measurement from repeatedly moving traffic between targets.

AWS documents InvocationThrottles as dependent on SDK retry settings. A retry-policy change can alter the metric without any change in customer demand. Record attempts separately, and avoid interpreting a raw throttle count as a direct measure of rejected user requests. Quality gates still apply before a metric-driven policy shifts traffic.

Key distinctions and trade-offs

The following policies differ in the evidence they consume:

Routing is also distinct from cascading. A router can select a target before inference; a cascade can invoke another model after evaluating an earlier result. See 2.2.9 for cascading. Likewise, failure-triggered fallback is not proof that the primary route was content-based or metric-driven.

Architecture / decision scenario

A support service receives short navigation requests, complex policy questions, and image-bearing requests through one application contract. Initially, it uses static assignments for each approved task class to establish quality and latency baselines.

For English text requests within a supported model family, the team evaluates intelligent prompt routing. It keeps image and other capability-constrained requests on separately validated paths. The gateway retains the common response schema, so callers do not change their integration when text routing changes.

Production evidence then shows sustained latency degradation for one target. The application consumes model-scoped InvocationLatency, checks dimensionless error metrics for pre-routing failures, and tracks InvocationThrottles separately. It shifts eligible traffic only to an alternative that passed the task’s quality gate. It does not ask the managed prompt router to consume those workload metrics; that feature does not support that control loop.

The final policy combines content eligibility, static assignments for constrained work, and measured preference among compatible alternatives. Each request records the route reason and policy revision, allowing the team to distinguish a task-class decision from an operating-health decision.

Common exam traps

• Assuming a common inference API guarantees equal model capabilities.
• Letting an untrusted caller bypass the target allowlist with modelId.
• Treating intelligent prompt routing as arbitrary cross-family routing.
• Claiming managed prompt routing reacts to application-specific runtime metrics.
• Calling CloudWatch metric publication an automatic model-routing mechanism.
• Allowing model-specific response shapes to escape the normalized contract.

What to remember

Keep one caller contract and validate each eligible adapter. Choose static routing for reviewed assignments, content-based routing for request differences, and metric-driven routing for measured conditions. Bedrock intelligent prompt routing has family, language, and performance-data boundaries. CloudWatch provides the signals; your application implements a bounded decision policy.

Knowledge check

References

• AWS, Converse.
• AWS, Understanding intelligent prompt routing in Amazon Bedrock.
• AWS, Monitor bedrock-runtime inference using CloudWatch metrics.