Debug a multi-agent production incident with trace evidence

Why it matters

The failing span in an incident is often a symptom; a root cause has to be supported by evidence before a repair is applied.

What to practise

Emit a synthetic checkout incident through OpenTelemetry and narrow it with three KQL stages: candidates, one trace’s detail, and a comparison of failed and successful cohorts. Save a sanitized snapshot to Blob Storage with configuration versions, model deployments, prompt hashes, and mocked tool responses, then replay it without side effects: the replay verifies prompt hashes and substitutes the mocked tool responses, but does not rerun the model. Change the prompt text and watch the hash diverge. Score each hypothesis as supporting, contradicting, or missing evidence. Then follow an Azure Monitor alert through an action group to Event Hubs, where a bounded consumer records a proposed containment action. No Foundry project or model is provisioned.

How it fits

Reconstruction, safe replay, and evidence-backed root cause are 3.3.9. The alert-to-Event Hubs path and its bounded, side-effect-free remediation are 3.3.4. Compare the cohort query with the per-step failure attribution in 3.1.7.