Set quality gates before model consumption

Learning objective

Define FM input quality thresholds for completeness, consistency, validity, and suitability.

Core idea

An input quality threshold is a release condition for data entering a foundation model (FM). It states what must hold, how you measure it, and what happens when it fails. A successfully parsed record can still be incomplete, contradictory, or unsuitable for the task. A model’s ability to produce fluent text does not remove these defects.

Separate mandatory conditions from tolerated defects. A missing contract identifier can prevent reliable attribution, while a missing optional contact number might have no effect on summarization. Your policy must express that difference before generation. Otherwise, an aggregate quality score can conceal the failure that matters most.

AWS Glue Data Quality provides Data Quality Definition Language (DQDL) rules for measurable dataset checks. It supplies mechanisms for many structural conditions; you still decide the business threshold and the routing policy. There is no universal AWS completeness percentage that establishes whether a dataset is safe for your use case.

How it works

Define the unit and denominator

Start with the unit you release: a record, document, partition, or complete batch. For every gate, record the population assessed, the condition, the threshold, and the failure action. For example, require every admitted document to have a source identifier, while allowing a small proportion of an incoming batch to be quarantined for missing optional metadata.

These are different policies. A batch with 99% complete identifiers still contains records without identifiers. Passing a dataset threshold does not grant those records permission to reach the model. Keep the batch acceptance decision separate from individual-record admission.

Use a representative set of accepted and rejected inputs to select thresholds. Include difficult cases such as truncated documents, blank strings, contradictory versions, and records from uncommon sources. The chosen boundary should reflect the cost of passing a defective input and the cost of rejecting a usable one. Document illustrative percentages as workload decisions, rather than service defaults.

Measure four dimensions separately

Map each dimension to an explicit check:

Express the chosen thresholds in a DQDL ruleset:

Rules = [
  IsComplete "source_id",
  Completeness "phone" > 0.8,
  ColumnValues "region" in ["EU", "US"] with threshold > 0.98
]

These illustrative thresholds require every source identifier, more than 80% complete phone values, and more than 98% region compliance. Completeness measures a dataset’s non-null ratio; with threshold tests the proportion satisfying a condition and works only with ColumnValues, ColumnDataType, and CustomSQL. Neither tolerated ratio admits each individual record.

Completeness concerns required information, not just column presence. ColumnExists checks whether a column exists; it does not establish that each row contains a usable value. DQDL also distinguishes NULL, EMPTY, and WHITESPACES_ONLY in value expressions. State which representations your input contract treats as missing instead of relying on a parser’s behavior.

Consistency requires a reference or a relationship. Two individually valid dates can contradict each other if the termination date precedes the start date. Two individually valid documents can conflict if both claim to be the applicable policy. A type check cannot resolve either conflict. Use a defined business relationship and retain the evidence behind its outcome.

Validity establishes conformance to a schema or rule, not truth. A syntactically correct account number can identify the wrong account. Combine value validation with the relevant reference checks when identity affects the task. Avoid normalizing a rejected value into an apparently valid value without recording the change.

Suitability asks whether the accepted material is useful for the intended operation. An approved, complete table might lack the units needed for a financial comparison. A readable policy might belong to the wrong product version. Define task-specific required metadata and reviewed content criteria. A custom check can test metadata; it cannot automatically establish the meaning or authority of every passage.

Turn results into release decisions

AWS Glue Data Quality supports checks in the Data Catalog and in extract, transform, and load (ETL) jobs. Catalog evaluation assesses cataloged data; ETL checks can identify and filter bad data before loading it. Choose the placement that enforces your release boundary, not merely the one that produces a dashboard.

Preserve the ruleset outcome for the batch decision and the policy version used to evaluate it. In Glue ETL, row-level results include DataQualityRulesPass and DataQualityRulesFail arrays and DataQualityEvaluationResult with Passed or Failed status. Route records using those results to accept, repair, quarantine, or reject. A row can pass while a dataset-level rule fails; a row result does not establish batch acceptance. Repair requires another validation pass: a transformation might fill a missing field while introducing the wrong value. Keep mandatory conditions as independent gates so a favorable average cannot compensate for their failure.

A repaired row returns for another validation pass, and an accepted row still waits on the batch acceptance decision.

Reassess thresholds when the input population or downstream task changes. A historical baseline can help detect operational changes, but it does not replace fixed business requirements. DQDL dynamic rules compare metrics with historical values through last() and are supported in Glue ETL, rather than Data Catalog quality evaluation. A consistently bad history is still bad.

Key distinctions and trade-offs

Dataset health versus record admission: Aggregate thresholds help decide whether a delivery is acceptable. Record checks determine which individual inputs can be consumed. Design both when partial acceptance is allowed.

Strict gates versus tolerated defects: Strict gates protect identity, authority, and required meaning. Tolerances preserve coverage for optional information. Increasing tolerance buys availability only when the missing information does not change the decision.

Quality versus model capability: A valid input can exceed a chosen model’s operating contract. Model capability and context eligibility belong to 1.2.1. Apply that contract after data quality checks; passing either boundary does not imply passing the other.

Thresholds also create operational work. Quarantine without an owner and a resolution path becomes an accumulating backlog. Assign ownership for failed conditions and make the policy’s terminal outcomes visible to downstream consumers.

Architecture / decision scenario

A procurement assistant summarizes supplier agreements. The team requires a source identifier, supplier identifier, contract version, and effective date for every admitted document. It tolerates missing optional telephone numbers because the summary does not use them.

The tabular metadata passes IsComplete for mandatory columns and ReferentialIntegrity for supplier identifiers against an approved supplier dataset. A CustomSql check identifies contradictory date relationships. An application check verifies that the selected agreement version is the one approved for the request.

One delivery has excellent overall completeness but contains agreements with no version. The pipeline quarantines those agreements and releases the remaining documents only if the delivery also meets its batch policy. It does not ask the FM to infer the missing version. After an operator repairs metadata, the repaired records pass through the same gates again.

Common exam traps

• Treating a column’s existence as proof that required values are complete.
• Using a dataset completeness threshold as an individual-record guarantee.
• Equating schema validity with factual correctness or task suitability.
• Letting a high overall quality score override a mandatory failed condition.
• Assuming historical quality thresholds establish business acceptability.
• Asking generation to repair missing authoritative identifiers without evidence.

What to remember

• Define the unit, denominator, threshold, and failure action for every gate.
• Use DQDL rules for measurable structural and relationship checks.
• Treat suitability as a task-specific contract that needs reviewed evidence.
• Separate batch acceptance from record admission and revalidate repairs.
• Quality results become protection only when the pipeline enforces them.

Knowledge check

References

• AWS, AWS Glue Data Quality
• AWS, DQDL rule types
• AWS, Data Quality Definition Language reference