Skip to lesson

AWSAWS Certified Generative AI Developer - Professional2026-100 Prerequisites

Validate outputs against model limitations

Completed by the knowledge check

Learning objective

Recognize model capabilities, nondeterminism, hallucinations, knowledge limits, and output limitations.

Core idea

For an engineer consuming generated results, a model response is a candidate output under a documented capability and resource contract. Fluency does not establish accuracy. A plausible answer can be unsupported, incomplete, outdated, or unsuitable for the task, even when the invocation succeeds.

Separate the failure mechanisms before choosing a control. Capability checks address whether the model can perform the required transformation. Repeated measurement addresses variation. Evidence checks address unsupported claims. Retrieval or approved tools can supply missing knowledge. Completion and schema checks address output boundaries. No single sampling setting or prompt replaces all of these controls. This lesson establishes the limits; the later quality and safety lessons define their production implementation.

How it works

Capability is eligibility, not guaranteed effectiveness

A model’s supported inputs, outputs, and features determine whether it is a candidate for a task. Bedrock’s FoundationModelDetails includes inputModalities, outputModalities, and responseStreamingSupported. These fields help identify basic capability boundaries, but they do not prove that a specific document, language, or business problem produces an acceptable result. See 0.1 for categories and 1.2.1 for model selection.

AWS’s Amazon Nova service card distinguishes a model’s broad capabilities from use-case effectiveness. It directs customers to measure effectiveness on their own representative inputs. A benchmark on general questions does not establish accuracy on your internal product vocabulary, photographed equipment, or ambiguous policy exceptions. Define what makes a result acceptable before interpreting a success rate.

The same service card identifies unsupported tasks and requires attention to consequential decisions. Do not infer that generating an answer means the model is designed for that use. If a task exceeds the documented intended use or your validated operating scope, route it to a supported process instead of treating a fluent completion as proof of eligibility.

Nondeterminism changes how you test

AWS describes generative models in the Nova service card as nondeterministic. Responses can vary, and small prompt changes or reordered examples can affect completions. The application must tolerate valid variation while detecting material changes in facts, decisions, and structure.

Temperature and related controls influence the sampling distribution; see 0.3. They do not create a general guarantee of identical output or factual accuracy. A lower-temperature response can consistently repeat an unsupported claim. More stable wording is not the same measurement as a lower error rate.

For a representative evaluation case, repeat invocations under a fixed model and request configuration. Measure whether required facts and decision outcomes remain acceptable, rather than requiring every sentence to match a golden string. If wording variation is allowed, evaluate it against the same rubric. If an exact identifier or amount must match, validate that field directly. See 5.1.2 for evaluation datasets and 5.1.5 for controlled comparisons.

Hallucinations are unsupported or inaccurate content

A hallucination can be a fabricated policy, citation, event, or relationship that the response presents as established information. It need not look unusual. The Nova service card warns that completions can sound more confident than their knowledge supports and that the models are not document-retrieval tools by themselves.

Judge a claim against authoritative evidence. Semantic relevance only establishes that a passage concerns the same subject; it does not prove that it supports the claim. Likewise, the model saying “I am certain” is text produced by the model, not an independently calibrated correctness signal.

Amazon Bedrock Guardrails provides contextual grounding checks that compare a response with a source and query. AWS documents separate grounding and relevance signals, with configurable thresholds from 0 to 0.99. The documented use cases include summarization, paraphrasing, and question answering; the page explicitly excludes conversational QA and chatbot use cases. Do not present the mechanism as a universal chatbot truth checker.

For a supported independent check through ApplyGuardrail, grounding_source and query qualifiers identify the reference and question. Use these signals as part of a calibrated acceptance policy, not as a guarantee that every accepted claim is true. See 3.1.6 for verification and 3.4.3 for confidence interpretation.

Knowledge boundaries require an evidence source

Training does not create a complete, continuously updated enterprise knowledge base. The Nova service card says its corpus does not cover every geography, time period, or domain, and does not define a training cutoff date. Other models can publish one: Bedrock’s Claude Sonnet 4.5 card lists April 2025 as its knowledge cutoff. The existence or absence of a date is model-specific.

A cutoff does not promise complete accuracy before that date. An omitted internal policy can be unknown even if it predates training. Conversely, an application can supply later evidence in a request. The relevant question is whether the invocation has reliable information for this particular decision.

Bedrock defines retrieval augmented generation (RAG) as retrieving information, adding it to the prompt, and generating with that context. This can supply approved domain information but does not eliminate retrieval errors or unsupported conclusions. Use an approved retrieval path for policies and an authorized tool for live account state, then validate the result against the returned evidence. See 1.5.6 for retrieval boundaries and 2.1.10 for tool contracts.

Output boundaries remain independent

A valid request can end with a refusal, a filter intervention, or an incomplete response. Converse returns stopReason values including max_tokens, guardrail_intervened, content_filtered, and model_context_window_exceeded. Inspect them before consuming the output; a successful HTTP response does not mean the requested business record is usable.

For eligible models, Bedrock structured outputs constrains JSON through outputConfig.textFormat in Converse. Its supported JSON Schema subset does not include numerical minimum and maximum constraints. Even supported shape constraints do not verify facts, entitlements, or business rules. Check those independently; see 3.1.4. A perfectly shaped invented refund amount remains an incorrect refund amount.

Key distinctions and trade-offs

Nondeterminism versus hallucination: Variation concerns differences between outputs. Hallucination concerns unsupported or inaccurate content. Either can exist without the other.

Knowledge cutoff versus coverage: A date describes one knowledge boundary. It does not certify all earlier facts or make private enterprise data available.

Grounding versus relevance: A response can answer the topic while contradicting the source, or faithfully repeat a source passage that does not answer the query. Both checks matter where supported.

Schema versus business validity: Structured output helps consumers parse records. Independent rules establish whether those records are suitable to act on. Avoid adding retries that merely regenerate the same unsupported conclusion.

Architecture / decision scenario

A support workflow drafts a warranty assessment from a photographed serial label, a policy document, and the customer’s account state. The team first verifies the chosen model supports the supplied modalities. It evaluates difficult labels and policy exceptions, including repeated runs with the same configuration.

The model cannot obtain live coverage from training. An approved account tool provides that state, while retrieved effective policy text supplies the rule. The workflow checks whether the draft’s claims match those inputs. A separate single-turn policy-summary function can evaluate the documented contextual grounding mechanism; the team does not assume its scope includes the entire conversation.

Before presenting a proposed assessment, the application inspects stopReason, validates any structured fields, and checks business rules against the source records. Missing evidence, conflicting repeated decisions, and truncated output produce review or clarification, rather than automatic approval. This separates capability, variability, hallucination risk, knowledge access, and output completion instead of attempting to solve all five with temperature.

Common exam traps

  • Equating fluent or confident language with accuracy.
  • Treating lower temperature as a correctness or repeatability guarantee.
  • Assuming all information before a cutoff date is known and reliable.
  • Treating a foundation model as a live enterprise database.
  • Assuming contextual grounding supports every conversational use case.
  • Treating schema compliance or successful transport as business validity.

What to remember

Recognize which boundary failed before choosing a remedy. Verify task capability, measure acceptable variation, supply authorized evidence, check consequential claims, and inspect output completion. Model-generated confidence and valid JSON do not replace independent evidence and application rules.

Knowledge check

References