Monitor and trace your generative AI agent

Why it matters

A longer prompt changes what every call costs and how long it takes; runtime telemetry shows by how much before you choose which version to keep.

What to practise

Get the Application Insights connection string from the Foundry project client, pass it to configure_azure_monitor(), and let OpenAIInstrumentor create a span for every chat completion. The script runs the same five test prompts under prompt versions v1, v2, and v3, each wrapped in a version span with per-test token and duration attributes. Compare total, prompt, and completion token counts under Monitoring › Resource usage in the Foundry portal, then read the local span tree to compare durations per test. Optionally, alert on total token count.

How it fits

Connecting tracing to the project’s Application Insights and attaching token attributes to application spans is 3.3.8. The token counts feed 3.3.7, and the optional alert is a first step toward 3.3.4. The script calls the model with each version’s system prompt, so it traces model calls rather than agent runs.