Skip to lesson

MicrosoftMicrosoft Certified: Multi-Agent AI Solutions Expert2026-100 Prerequisites

Correlate logs, metrics, and OpenTelemetry traces

Completed by the knowledge check

Learning objective

Interpret operational logs, metrics, traces, spans, and basic OpenTelemetry concepts.

Core idea

Different operational questions require different evidence. Metrics expose population-level changes, logs record contextual events, and traces follow operations through related spans. Combine the signals to explain a failure.

For an architect preparing for AI-500, distinguish the telemetry representation from the system that stores it. OpenTelemetry supplies instrumentation, collection, and export mechanisms. Azure Monitor supplies a data platform and analysis tools, including Application Insights for application tracing. OpenTelemetry is not itself the database or visualization backend. A configured exporter is also not proof that the spans you need were produced.

How it works

Read logs as contextual events

A log is a timestamped record. It can contain structured fields or free-form text. Operational logs can describe a rejected request, a completed tool call, or an exception. They explain an event at one component; without correlation, they do not establish the entire request’s path through a distributed application.

Azure Monitor Logs stores records in a Log Analytics workspace. You query them through Log Analytics using Kusto Query Language (KQL). Azure Monitor distinguishes this storage and query platform from sources such as the Azure Activity log and resource logs. The source answers who emitted the event; the platform answers where you query the collected records.

Record the operation name, outcome, and correlation identifiers as application fields, not a universal Azure schema. An explicit outcome distinguishes completion from rejection or failure.

Keep logs bounded to their purpose. The Foundry tracing guidance warns that user input, outputs, and tool arguments can contain sensitive information. Redact or minimize that content before telemetry emission, and do not put secrets, credentials, or tokens into span attributes or logged tool arguments.

Read metrics as aggregate measurements

Azure Monitor metrics are numerical measurements identified by time, name, value, and labels. Azure Monitor stores them in a time-series platform suited to trend detection and alerting. Metrics Explorer supports exploration, while metric alerts evaluate conditions. A sudden change tells you where to investigate; it rarely supplies a complete causal explanation by itself.

OpenTelemetry uses a MeterProvider to create Meter instances, which create metric instruments. A Counter accumulates increases, such as completed-request counts. An UpDownCounter supports increases and decreases, such as changes in active work. A Gauge measures a value at the time it is read. A Histogram aggregates distributions, such as request durations.

Choose the instrument according to the measurement. A Histogram is useful when the question concerns the distribution of latency rather than only its average. A Counter is appropriate for accumulating failures, while a Gauge fits a sampled queue-depth value. The names you assign to custom instruments belong to your application’s telemetry contract; do not mistake illustrative names for built-in Azure metrics.

Define the population and time interval before interpreting a chart. An average over all requests can hide a slow tool path. A failure count without the total request count cannot tell you whether the failure rate increased. Decide which dimensions identify useful groups, such as operation or deployed version, rather than relying on a global number for every diagnosis.

A trace describes the path of an operation. A span is one unit of work with a start time, end time, context, and associated information. Spans in one trace share a trace ID, while each span has its own span ID. A child span identifies its parent; a root span has no parent.

OpenTelemetry’s trace concepts illustrate these relationships through trace_id, span_id, and parent_id. These conceptual example fields are not a promise about every backend’s stored column names. In an Azure query, use the documented schema of the actual table rather than pasting illustrative trace JSON as if it were the storage format.

Span attributes describe the operation. Span events mark significant points in time within it. For example, a retry-scheduled event records when that decision occurred, while an attribute can identify the operation’s target. Span links express relationships to other spans when a simple parent-child structure does not capture the relationship adequately.

Span kind distinguishes execution roles: Client for an outgoing remote call, Server for an incoming remote call, Internal for work inside a process, and Producer or Consumer for asynchronous job creation or processing. Here, a synchronous remote interaction does not mean the programming language cannot use async and await.

Span status uses Unset, Error, or Ok. The status describes the instrumented operation, not a general guarantee about the business result. A model request can complete without a transport error while producing an answer that fails application validation. Record and evaluate that outcome separately instead of equating a completed span with a correct answer.

Propagate context across services

A distributed trace needs context propagation between callers and receivers. OpenTelemetry’s default propagator uses W3C Trace Context. For HTTP, the traceparent header carries trace ID, parent span ID, and trace flags. The sender injects context into the carrier; the receiver extracts it and associates its work with the incoming context.

Instrumentation libraries often perform propagation automatically; see 1.2.7 for propagation through agent and tool calls. When an unsupported integration requires manual handling, use the Propagators API and the protocol’s metadata carrier. Do not invent an unrelated request identifier and expect the tracing backend to infer the parent relationship.

OpenTelemetry SDKs can also add active trace and span context to log records. That correlation lets you move from a trace’s slow operation to the relevant event details. A trace ID groups execution evidence; it is not an authorization credential or an agent conversation ID.

Connect instrumentation to an Azure backend

OpenTelemetry includes APIs, SDKs, exporters, semantic conventions, and the Collector. The Collector receives, processes, and exports telemetry. Those roles remain separate from the storage backend. Follow the path from instrumentation to export and ingestion when evidence is missing.

Foundry’s tracing setup recommends server-side tracing after connecting Application Insights to the project; out-of-the-box traces cover the past 90 days after tracing is enabled. For Python client-side instrumentation, install opentelemetry-sdk and the Azure SDK tracing plugin azure-core-tracing-opentelemetry alongside azure-ai-projects and azure-identity. The setup guide requires Microsoft Entra ID for using the project endpoint; without that configuration, use the Application Insights connection string for telemetry. Microsoft states that tracing is generally available for prompt and hosted agents, while workflow and external-agent tracing are in preview.

Key distinctions and trade-offs

Metric versus trace: A metric summarizes a population. A trace provides execution structure for an operation. Use metrics to locate a symptom and traces to inspect the relevant path.

Log versus span: A log records an event; a span has a duration and relationships. Correlation connects them without making them interchangeable.

Instrumentation versus backend: OpenTelemetry emits and transports evidence. Application Insights and Azure Monitor store and expose it. Configuring the latter cannot reveal custom application work that has no instrumentation.

Architecture / decision scenario

A support application’s latency chart rises after deployment. Metrics Explorer confirms that the increase concerns one operation group. The team opens a representative trace in Application Insights and finds a long storage-call span beneath the agent request. Its correlated log records a retry outcome, while the model-call span remains short.

The team checks propagation through traceparent, because some requests initially appeared as disconnected traces. It uses opentelemetry-sdk with azure-core-tracing-opentelemetry for client-side instrumentation and adds a span around custom validation. A Histogram supplies latency distribution, a Counter tracks failures, and a sampled Gauge tracks queue depth. Log Analytics queries provide event context. The investigation separates a slow dependency from model behavior and keeps sensitive document text out of telemetry.

Common exam traps

  • Treating OpenTelemetry as the telemetry storage backend.
  • Expecting an aggregate metric to explain one request’s complete path.
  • Confusing trace ID, span ID, and conversation ID.
  • Assuming a log line automatically has a parent-child tracing relationship.
  • Treating a successful span as proof of a correct model answer.
  • Assuming server-side agent tracing covers all custom application code.

What to remember

Use metrics for trends, logs for contextual events, and traces for related timed work. Spans and propagated context make the execution path inspectable. Verify instrumentation, export, ingestion, and query access separately, and keep telemetry outcomes distinct from business correctness.

Knowledge check

References