Generate Synthetic Test Data for Your Prompt Template

Why it matters

Real inputs are often missing, too few, or off-limits for privacy reasons. An architect has to know what synthetic cases can stand in for and what they cannot.

What to practise

Extract the {{variables}} from a customer support prompt template, and have Claude plan each variable’s realistic source, length, format and tone in a <planning> block before it writes a value for each one. Supply a golden example and check that new cases come from the same distribution while still adding signal. Edit the planning text to steer later cases, for example toward numbered policy documents. The notebook sends that edit as a prefilled assistant turn, which its default model, Claude Sonnet 4.6, rejects with a 400 error; put the edited planning in the user turn instead. Decide who writes the golden answers.

How it fits

This is the synthetic slice of 4.2.1. Compare the cases you generate with production traffic: they add coverage, but they do not tell you how the system performs on real users. A case you keep becomes a maintained artifact under 4.2.4’s versioning and held-out rules.