Configure Data Pipelines with AWS Glue

Why it matters

Preparing data for a GenAI workload requires a working processing pipeline. This exercise provides general data-engineering practice for subsection 1.3 through dataset integration, orchestration, and output validation.

What to practise

Crawl two S3 datasets and join their cataloged tables in a Glue Visual ETL job. Combine the crawler and job in a trigger-driven Glue Workflow, and use job bookmarks for incremental processing. Visualize processed data in an interactive notebook and inspect the output with Athena.

How it fits

No 1.3 Bite teaches Glue ETL itself. For a GenAI workload, the parts that carry over are job bookmarks, which process new data incrementally, and checking the output with Athena before anything downstream consumes it.