AWSAWS Certified Generative AI Developer - Professional2026-102.3 Design and implement enterprise integration architectures

Centralize inference policy behind a gateway

Completed by the knowledge check

Learning objective

Centralize authentication, rate limits, governance, routing, and observability behind a GenAI gateway.

Core idea

A GenAI gateway is an application architecture boundary where inference requests acquire a common policy before reaching a model. Amazon API Gateway can provide the API entry point, authentication integration, request throttling, and access logs. Your backend supplies model selection, token budgets, approved routing, and release policy.

For an architect serving several enterprise applications, the benefit is consistent enforcement and evidence. Clients request an approved task or capability instead of selecting arbitrary models and destinations. The gateway decides whether to admit the request, which eligible route to use, and how to account for its result.

Centralization works only when downstream access cannot bypass the boundary. Give the gateway’s workload the required inference authority and review direct client permissions separately. Per-user tool and document authorization remains necessary; see 2.3.7.

How it works

Authenticate before selecting a route

An HTTP API JWT authorizer validates the token and route scopes before the integration runs; see 2.3.7.

Use the validated caller context to derive the application caller and tenant. Map the requested capability to an authorized product operation before model selection. A caller allowed to summarize public material should not gain access to a confidential workflow by asking for a more capable model.

API Gateway warns that JWT access tokens and other JWT types do not have a universal distinguishing mechanism. Configure scopes or provider-specific issuer and audience choices appropriate to the intended access-token flow. Do not assume that accepting any token from a familiar provider is sufficient API authorization.

REST API usage plans and API keys serve a different purpose. They identify clients for metering and request-limit targets. AWS explicitly says not to use API keys for authentication or authorization. If a client presents an API key, retain an actual authorization mechanism rather than treating possession of the key as proof of user identity.

Separate request throttling from inference budgets

API Gateway uses a token bucket for request throttling. The tokens in this algorithm represent requests, not language-model tokens. For HTTP API route settings, ThrottlingRateLimit sets the rate target and ThrottlingBurstLimit sets the burst target. Excessive traffic can receive 429 Too Many Requests.

AWS describes API Gateway throttles and quotas as best effort. REST API usage-plan limits are not hard ceilings, and clients can exceed them. Use these controls to reduce overload, but do not rely on them as a strict financial cap or a confidentiality boundary.

AWS recommends AWS Budgets for cost monitoring and AWS WAF for API request management as complementary controls; neither is a hard spending cap.

Add application-owned accounting for model-token and workflow budgets. A short request and a long-context request can each consume one API request while generating very different inference work. Maintain a budget reservation before dispatch and reconcile it with the returned usage. Define what happens when the budget is exhausted: reject, defer, or choose an already approved cheaper route.

Keep retries inside the same budget. Several backend attempts after one admitted request must not appear as one inexpensive call in tenant accounting. The application ledger needs attempt identity, the selected route, and the usage outcome; this accounting is a design responsibility, not a feature supplied by an API Gateway usage plan.

Use Bedrock operational metrics to diagnose downstream pressure. The AWS/Bedrock namespace includes InputTokenCount, OutputTokenCount, InvocationThrottles, and InvocationLatency for the documented bedrock-runtime endpoint. These service metrics support operations; aggregate metrics alone are not a per-user admission ledger.

Govern and route through approved configuration

Keep the allowed model routes in versioned application configuration. For each task, define eligible model identifiers, processing locations, interface requirements, fallback order, and acceptance policy. Reject unsupported task-route combinations before inference rather than letting arbitrary client parameters become a production routing policy.

Enforce required Bedrock Guardrails independently of gateway code. On the workload role, use an explicit deny with StringNotEquals on bedrock:GuardrailIdentifier against the approved guardrail ARN and numeric version. AWS documents this enforcement for InvokeModel, InvokeModelWithResponseStream, Converse, and ConverseStream: requests without the required guardrail are denied even if other policies grant inference. See 3.1.1 and 3.1.2 for guardrail configuration and application.

Bedrock cross-Region inference profiles define model and destination-Region sets. Geographic profiles route within their geography; global profiles route across supported commercial Regions. Select them only when the processing policy permits their destinations. A locally hosted gateway does not make a cross-Region inference route local. See 2.3.8 for verification evidence and 1.2.4 for inference-route choices.

Centralize model fallback under one bounded policy. A downstream throttle can be a reason to defer or select an eligible fallback, but it is not permission to leave the approved geography or reset the request budget. See 1.2.5 for bounded fallback and terminal outcomes.

Govern changes to this configuration with the release gates in 2.3.9. A route policy change can alter privacy, cost, or output quality even when the gateway executable is unchanged. Record the policy revision with each decision so operators can relate a behavior change to the release that introduced it.

Correlate gateway and model evidence

API Gateway HTTP API access logs can include $context.requestId, $context.routeKey, $context.status, and $context.responseLength in a CloudWatch Logs destination. Use the request identifier as the starting point for an application correlation record across admission, authorization, routing, and downstream attempts.

Add application fields for the selected model route, release revision, tenant identity, budget decision, and terminal outcome. These fields are your logging design, not automatic API Gateway context variables. Prefer a minimal structured record; avoid logging bearer tokens or complete sensitive prompts merely to obtain request correlation.

Bedrock’s InvocationLatency measures time until the last token is received. TimeToFirstToken applies to the documented streaming operations. Distinguish these from client-visible latency, which also includes gateway admission and application processing. Inspect InvocationClientErrors, InvocationServerErrors, and InvocationThrottles separately rather than grouping every failed request as a model outage.

The metrics have important counting boundaries. Invocations includes accepted requests that later fail but excludes throttled requests and requests rejected before processing. Errors can be published without ModelId when Bedrock cannot determine the model. Querying only model-specific metrics can therefore miss some errors.

Compare the application admission count with downstream attempts and terminal outcomes. A spike in gateway denials with stable Bedrock metrics points to a different layer than a spike in accepted requests followed by downstream throttles. Correlation makes that distinction actionable without exposing request content.

Key distinctions and trade-offs

API entry point versus model-policy engine: API Gateway supplies API controls. The backend implements the workload’s model routing and token-accounting decisions. Calling the combination a gateway does not make all its policies managed-service features.

API Gateway admits the caller; the backend authorizes the task, reserves the budget before dispatch and reconciles it with the returned usage.

Request tokens versus model tokens: API Gateway token-bucket tokens count requests. Bedrock token metrics describe inference input and output. Neither is interchangeable with a strict per-tenant budget reservation.

Usage-plan key versus authenticated identity: An API key supports metering. Verified tokens or another supported authorizer establish API access.

Central consistency versus shared failure: One policy boundary reduces divergent client behavior but adds a shared dependency. Design recovery routes with the same authorization, budget, and residency rules rather than opening a direct bypass during an outage.

Architecture / decision scenario

Three departments use a shared assistant gateway. Support asks for short summaries; legal submits long documents; engineering needs streamed answers. All have valid access tokens, but their approved tasks and budgets differ.

The HTTP API validates tokens and scopes. The backend derives tenant context, authorizes the task, reserves its model-token budget, and selects a route from approved configuration. API Gateway request throttling reduces overload, while backend accounting prevents a department from evading its budget through repeated large requests.

Legal’s request cannot use a global profile because its processing policy forbids it. Engineering’s stream is observed with TimeToFirstToken and client-visible timing. A downstream throttle is correlated with the gateway request and charged to the shared attempt budget before any eligible retry. An exhausted request terminates explicitly rather than bypassing the gateway or selecting an unapproved model.

Common exam traps

Using an API key as the authentication mechanism.
Equating API request-rate limits with model-token budgets.
Treating usage-plan quotas as guaranteed spending ceilings.
Assuming a local gateway makes every model route local.
Letting retries or fallback bypass the original policy.
Reading only ModelId metrics and missing unclassified errors.

What to remember

Authenticate and authorize the task before route selection.
Separate request throttles, model capacity, and tenant accounting.
Govern routes as behavior-changing release configuration.
Correlate gateway admission with every downstream attempt.
Apply the same policy to recovery paths and normal requests.