AWSAWS Certified Generative AI Developer - Professional2026-101.3 Implement data validation and processing pipelines for FM consumption
Match preprocessing compute to work units
Completed by the knowledge check
Learning objective
Choose SageMaker Processing for batch preprocessing versus Lambda for event-driven transformations.
Core idea
Choose compute by the unit of work it must complete. A short transformation of one arriving record has different operating requirements from a reproducible transformation of a large dataset. Arrival pattern matters, but runtime, resource requirements, dependencies, and output validation determine whether the chosen execution model fits.
Amazon SageMaker Processing runs data processing scripts in managed infrastructure using built-in or custom containers. AWS Lambda executes function code with a configured timeout and resource configuration. Use Processing for dataset-oriented work that benefits from a job boundary and explicit compute selection. Use Lambda for bounded event-oriented transformations that fit its execution contract.
An event can start a batch job. That does not make the batch transformation a good function workload. Likewise, a scheduled function remains suitable if the work it performs is small and bounded. Separate the trigger from the compute that processes the data.
How it works
Define the workload envelope
Measure the largest supported input, the required libraries, peak memory use, CPU demand, processing duration, and output size. Include malformed inputs and slow dependencies in the assessment. A successful test on a small file establishes little about the largest production object.
Define what counts as completion. For a record transformation, completion might mean a validated normalized record has been written. For a dataset transformation, completion might mean all required partitions and a validation report exist. A compute success signal should not be the only evidence that consumers can safely use the outputs.
Include recovery in the unit of work. If replaying a single event is inexpensive, a function can provide a useful boundary. If the transformation compares records across a large dataset or must produce one reconciled release, a job boundary often better represents what the application needs to accept or reject.
Configure a Processing job explicitly
SageMaker Processing supports built-in data processing containers and custom containers. After submission, SageMaker AI launches the compute instances, runs processing, and releases resources on completion. Outputs are stored in the specified Amazon S3 bucket. The service manages the infrastructure lifecycle; your code still owns transformation correctness.
The CreateProcessingJob API makes the contract visible. AppSpecification selects the container image and its execution settings. ProcessingInputs configures inputs downloaded into the processing container. ProcessingOutputConfig describes outputs. ProcessingResources identifies compute instances and storage volumes, and RoleArn identifies the role that SageMaker assumes for the job.
Use StoppingCondition to bound how long the job can run. More available runtime is a capacity choice, not permission to leave a runaway transformation unbounded. Tie the job configuration to the input release and transformation version so a replay has an identifiable execution contract.
ProcessingS3Input.S3DataDistributionType controls distribution: FullyReplicated sends all data to every processing instance; ShardedByS3Key downloads one shard to each instance. With FullyReplicated on N instances, code that processes every local input processes the whole dataset N times unless it partitions the work. Match application logic to the selected distribution. Measure the complete job, including data movement and output production, rather than assuming that more workers always shorten the critical path.
Bound Lambda transformations
Lambda’s timeout setting bounds function execution. AWS documents a maximum of 900 seconds, or 15 minutes, for the ordinary function configuration. Its timeout page also documents a Managed Instances exception: asynchronous and event source mapping invocations, except Amazon MQ and Amazon DocumentDB, can allow up to 5,400 seconds. Do not present 15 minutes as universal across every Lambda execution mode.
Lambda function memory ranges from 128 MB to 10,240 MB, with CPU proportional to configured memory; /tmp storage ranges from 512 MB to 10,240 MB. Compare peak memory and temporary-file requirements with these limits alongside runtime.
The exception does not remove the workload decision. Verify the selected execution mode and invocation path, then assess whether its resource and runtime contract fits the work. A longer timeout alone does not provide dataset reconciliation, reproducible partitioning, or validation of a complete output release.
For bounded transformations, test with inputs near the maximum supported size and choose a timeout that accommodates measured variability. AWS warns that timeouts close to average duration increase the risk of unexpected failures as data size, dependency performance, and computational complexity vary.
Lambda’s memory configuration also affects available CPU. AWS recommends performance testing to choose memory size. A low memory setting can lengthen a CPU-heavy transformation, while a larger setting can change its runtime. Measure the workload rather than treating memory as only a storage requirement.
Write idempotent code. AWS recommends validating events and gracefully handling duplicates; event source mappings can process records more than once. Design the output identity around the source and transformation version, and make replay preserve the intended result. A duplicate event should not produce a second accepted artifact or overwrite a newer transformation’s output.
Separate submission from completion
A Lambda function can perform quick admission checks and submit a Processing job when the data requires batch work. That function’s success means submission succeeded; it does not mean the batch output exists or passed validation. Represent those milestones separately in the application.
Keep processing outputs separate from released artifacts until the checks finish. If a job or function fails after writing partial output, downstream consumers should not interpret the partial files as a complete release. Use application-controlled release metadata to identify the accepted output set and its validation evidence.
Submission, batch output, and passed validation are separate milestones, and partial output never becomes a release.
Observe execution and data outcomes independently. Lambda Duration, Errors, and Throttles help diagnose function behavior, while application checks establish output acceptance; see 1.3.2. For either compute path, retain the input identity, execution configuration, and validation outcome needed to investigate a failed release.
Key distinctions and trade-offs
Event-driven versus short-running: An event is an arrival mechanism. It can initiate long-running work on another service. Choose Lambda for the bounded function stage, rather than forcing all event-triggered work into one invocation.
Managed function versus managed job: Both reduce infrastructure management. Processing exposes an explicit container, compute, input, and output job contract. Lambda fits a function-oriented contract with timeout and resource settings.
Latency versus release reproducibility: Small independent transformations can favor event-by-event handling. Dataset-wide reconciliation can favor a batch release. If a workflow needs both, use separate work units with a clear handoff.
Cost comparisons also need measured work units. A function’s processing duration and memory configuration do not describe a Processing job’s complete operating envelope. Compare the cost of producing an accepted artifact, including repeated work and validation, rather than choosing from the number of services in a diagram.
Architecture / decision scenario
A knowledge ingestion pipeline receives small metadata updates and occasional large archive deliveries. The metadata updates need schema validation and a short normalization step. The archive requires custom parsing libraries, cross-file reconciliation, and one validated dataset release.
Lambda handles the small updates using idempotent output identities. For an archive event, Lambda validates the delivery metadata and submits a CreateProcessingJob request with pinned execution configuration and separate output locations. The archive transformation runs in Processing.
The application records submission, execution completion, and release validation as different states. If a Processing job fails after writing some files, those files remain unreleased. Retrying the delivery creates evidence tied to the same source version and the selected transformation, rather than silently mixing partial outputs from different attempts.
Common exam traps
- Choosing Lambda merely because an S3 event starts the work.
- Using average input duration to certify maximum-input processing.
- Treating a successful job submission as completed preprocessing.
- Assuming additional Processing instances automatically parallelize custom code.
- Treating timeout increases as a substitute for workload decomposition.
- Stating the ordinary 15-minute limit without considering documented execution-mode exceptions.
What to remember
- Define the work unit and completion contract before choosing compute.
- Use Processing for an explicit dataset-oriented execution contract.
- Use Lambda for transformations that fit the selected function execution mode.
- Test resource configuration and make duplicate processing idempotent.
- Release outputs only after complete validation.
Knowledge check
An S3 event announces a large archive delivery. The archive needs custom parsing libraries, cross-file reconciliation, and one validated dataset release. Which design matches the work unit?
A Processing job runs on four instances with S3DataDistributionType set to FullyReplicated. The custom script processes every file it finds locally, and the output contains each record four times. What change fixes the duplication?
ShardedByS3Key , or partition the work in the scriptStoppingCondition so that every instance can finishFullyReplicated and deduplicate in the release stepFullyReplicated sends all the input data to every instance, so code that processes every local file processes the whole dataset once per instance. ShardedByS3Key downloads one shard to each instance; otherwise the script must partition the work itself. More instances only multiply the repetition.A Lambda function normalizes uploaded metadata files of bounded size. Its timeout was set just above the average duration measured on typical files, and it now times out on the largest supported files. How should the team choose the timeout?
A Lambda function submits a Processing job successfully. The job writes part of its output to S3 and then fails. What should downstream consumers see?
A Lambda function consumes records through an event source mapping. After a redelivery, the same source record produces a second accepted artifact beside the first. What design change prevents this?