AWSAWS Certified Generative AI Developer - Professional2026-101.3 Implement data validation and processing pipelines for FM consumption
Preserve evidence through multimodal preprocessing
Completed by the knowledge check
Learning objective
Build multimodal text, image, audio, and tabular preprocessing pipelines with provenance and validation checks.
Core idea
A multimodal pipeline should normalize the contract around its outputs without erasing the differences between its inputs. Text offsets, document locations, image regions, audio intervals, and table coordinates carry different kinds of evidence. A single flattened text field cannot preserve all of them.
Provenance answers which source produced an artifact and which transformation created it. Validation answers whether that artifact satisfies the next stage’s contract. Keep both at each handoff. A traceable artifact can still be wrong, and a plausible artifact with no source mapping is difficult to review or reproduce.
Use modality-specific branches that converge on an application-defined artifact envelope. Preserve raw source versions, retain extraction evidence, record transformation configuration, and release outputs only after the required checks pass. The envelope is your design contract, not an AWS service’s universal multimodal schema.
How it works
Anchor each branch to a source version
Store source objects with identities that survive processing. Amazon S3 Versioning keeps multiple versions of objects and assigns unique version IDs to new objects written after versioning is enabled. Versioning is disabled by default, so explicitly enable it when object-version identity is part of the provenance design.
Record the bucket, key, and version ID used by the transformation. A key alone can refer to different bytes after an overwrite. AWS also documents that objects existing before versioning is enabled retain a null version ID until subsequent modifications create new versions. Do not retroactively claim that enabling versioning gave every old input an individually versioned identity.
Attach a run identifier and transformation version to each derived artifact. Record the configuration needed to explain the result, including parser settings and applicable quality policy. Source identity and transformation identity serve different purposes: the same input can legitimately produce different outputs under different transformation versions.
Keep raw sources separate from normalized outputs. Normalization can remove layout, change characters, or convert data types. Preserving the source lets a reviewer distinguish an intentional transformation from accidental loss. Versioning supplies recoverable source variants; it does not automatically create the relationships between sources and derived artifacts.
Validate within each modality
A text branch checks encoding, required content, and source metadata, then records how normalized text maps back to the original. If a transformation removes headers or joins lines, preserve enough mapping information to locate the original evidence. Application code implements these mappings; a successfully decoded string does not prove that its meaning survived.
A document-image branch validates that the selected extraction or understanding interface accepts the input, then preserves the source representation and output evidence. With Textract document analysis, retain Block relationships, pages, and locations instead of keeping only concatenated words. Service selection belongs to 1.3.3.
A photograph branch validates the consuming interface separately from document extraction. For a Bedrock model through Converse, ImageBlock.format is required and accepts png, jpeg, gif, or webp. Check the actual representation, record crop or resize decisions, and retain the source mapping. Model support for image blocks belongs to 1.2.1.
An audio branch validates the actual media encoding and sample rate before transcription. Transcribe’s batch and streaming paths have different format contracts, and a provided sample rate must match the actual recording. Retain word start and end times and confidence from transcription output. A summary derived from the transcript should still be traceable to the relevant recording interval.
A tabular branch preserves column names, types, units, row identity, and reference relationships. Apply the DQDL rules from 1.3.1 to measurable conditions, while preserving units and row identity through conversion.
Validate again after transformations. A parser can produce the expected schema while dropping pages. A numeric conversion can preserve the value while losing its unit. A text normalization can preserve readable words while changing a reference offset. Define checks for what each stage can lose, rather than repeating one generic non-empty-output check everywhere.
Converge without discarding evidence
Give every accepted artifact common metadata: source identity, modality, transformation version, validation outcome, and evidence locator. Keep modality-specific evidence beside that metadata. A document locator can identify a page and region; an audio locator can identify a time interval; a tabular locator can identify a row and column.
If an artifact combines multiple sources, record all contributing identities and mappings. Do not assign the merged output only the identity of whichever input happened to arrive first. Cross-source validation should also check that the sources describe the same case, version, or reporting period before combining their content.
Treat extraction confidence as one validation input, not the whole admission decision. A high-confidence transcript can describe the wrong case. A correctly extracted table can belong to a superseded report. The quality and suitability policy from 1.3.1 determines whether the evidence is admissible for the task.
Make retries preserve the release contract
AWS Step Functions provides Retry and Catch fields for supported workflow states. A retrier names matching errors and can set IntervalSeconds, MaxAttempts, and BackoffRate. Catchers route errors to recovery states after applicable retries fail. Use these mechanisms to make transient processing failures and terminal outcomes explicit.
A quality rejection is usually a data decision, not a transient execution failure. Repeating extraction on the same unsupported input does not make it valid. Route failed validation to repair, quarantine, or rejection according to the application policy. Reserve retry budgets for failures where another attempt can plausibly succeed.
Preserve original failure evidence when an attempt is repaired or rerun. Define output identity and deduplication at the application layer so a replay cannot publish multiple copies or combine artifacts from incompatible attempts. Stage outputs separately and publish release metadata only when the required branches and cross-source checks have completed.
The workflow should distinguish source receipt, extraction completion, validation completion, and artifact release. These milestones make partial success visible. A transcript completing while a required table branch fails is not a complete multimodal release.
Key distinctions and trade-offs
Common envelope versus common representation: Common metadata makes artifacts manageable. Modality-specific representations keep the evidence that gives content meaning. Preserve both rather than flattening every modality into text.
Provenance versus correctness: Version IDs and transformation records support attribution and reproduction. They cannot establish whether an extracted claim is correct or suitable. Quality gates remain necessary.
Retry versus repair: Retry repeats work after a potentially transient failure. Repair changes defective input or configuration, then requires another validation pass. Keeping these outcomes separate prevents endless retries on bad data.
Retaining evidence increases storage and metadata work, but the alternative makes investigation depend on guessing which transformation produced a result. Select the evidence needed for the workload’s review contract and manage source retention deliberately. Do not confuse retaining every intermediate file with maintaining useful provenance relationships.
Architecture / decision scenario
A maintenance assistant consumes technician notes, equipment photographs, recorded interviews, and sensor exports. Every source is associated with an equipment identifier and inspection period. The application records S3 object versions and creates separate modality branches.
The audio branch preserves transcript timing; the tabular branch checks types, required identifiers, and reference relationships with Glue rules. The photograph branch verifies a Converse-compatible format and preserves the original image and crop mapping before model interpretation. The notes branch checks encoding, required equipment metadata, and content, then preserves offsets through text normalization. Each branch emits validation evidence and common artifact metadata.
A cross-source check finds that the sensor export belongs to an earlier inspection. The workflow quarantines the combined release rather than letting a plausible summary conceal the mismatch. The corrected export runs through validation again. The released artifacts identify all contributing sources and transformations, so a reviewer can trace a recommendation back to its evidence.
Common exam traps
- Treating a source key as a stable identity after overwrite.
- Assuming S3 Versioning automatically records derived-artifact lineage.
- Flattening all modalities while expecting coordinates and timing to survive.
- Validating inputs but skipping checks after lossy transformations.
- Retrying quality rejection as though it were a transient service failure.
- Publishing a partial multimodal output as a complete release.
What to remember
- Preserve source versions and transformation identity at every handoff.
- Validate the failure modes specific to each modality and transformation.
- Converge on common metadata while retaining modality-specific evidence.
- Use explicit retry and recovery policies for execution failures.
- Release only a complete, validated, traceable artifact set.
Knowledge check
A multimodal pipeline merges transcript text, photograph descriptions and table values into one text field per case. Reviewers can no longer find where a statement came from in the original sources. Which artifact contract restores reviewable evidence?
An auditor must reproduce an artifact produced months ago. Since then, its S3 source key has been overwritten twice and the parser has been upgraded. What should every derived artifact have recorded?
A normalization step converts table values to numbers. The output passes schema validation, but the values have lost their units. Which validation policy catches this defect?
A Step Functions extraction state has a Retry that matches all errors with MaxAttempts set to 5. When a file fails validation because its format is unsupported, the workflow retries extraction five times and then fails. How should error handling change?
MaxAttempts and BackoffRate , so that the file gets more time to passIntervalSeconds so that the failure appears soonerCatch to route validation failures to quarantineCatch , so that every error fails the execution immediatelyIn one inspection case, the transcript branch completes. The required sensor-table branch fails cross-source validation because the export belongs to an earlier inspection. What should the workflow publish for the case?