NVIDIANVIDIA-Certified Professional: Agentic AI2026-108.2 Track logs, errors, and anomalies for root cause diagnosis.
Join service logs to workflow spans
Completed by the knowledge check
Learning objective
Correlate OpenTelemetry spans, structured logs, and request IDs across services.
Core idea
A timestamp tells you when an event was recorded. A request ID tells you which request it concerns. A trace tells you how work in different services relates. Production diagnosis needs all three: concurrent requests can produce similar messages at almost the same time, and one workflow can issue several tool and inference calls.
OpenTelemetry supplies context propagation and log correlation. NeMo Agent Toolkit supplies workflow telemetry exporters. NVIDIA NIM supplies structured logging and inference-header forwarding. Connect those mechanisms deliberately; enabling each independently does not prove that their records join into one execution. See 4.3.3 for using the resulting trace to investigate a loaded critical path. Here the decision is whether you can reconstruct that path at all.
How it works
Preserve causal context across each service boundary
OpenTelemetry propagation injects context into an outgoing carrier and extracts it at the receiving service. For HTTP, the W3C traceparent header carries the trace ID, parent span ID, and trace flags alongside a version. The receiver creates its own span in that trace, using the sender’s span as its parent. Each operation has its own span ID; copying one span ID into every downstream record would erase the distinction between operations.
Use instrumentation libraries where they support the transport. At a custom tool gateway, verify that the client injects context and the server extracts it. The OpenTelemetry Propagators API provides injection and extraction for such boundaries. If a worker receives a job through another carrier, preserve context in that carrier’s metadata and restore it before starting the processing span. A job payload containing a request ID alone does not implement propagation.
Configure workflow span export as in 4.3.3; here, verify that those spans can be joined to logs from each service.
Exporting workflow spans is one part of the path. Confirm that separately deployed services participate in the same trace rather than assuming the workflow exporter instruments every remote process.
Make structured logs joinable
The OpenTelemetry log data model defines Timestamp, ObservedTimestamp, TraceId, SpanId, SeverityText, Body, Resource, and Attributes. Timestamp describes event time; ObservedTimestamp describes when collection observed it. Comparing only collection times can make a buffered log appear to occur after its consequence.
TraceId and SpanId are optional in the model. OpenTelemetry SDKs can inject the active trace and span IDs into request-processing log records. If SpanId is present, the specification says TraceId should also be present. A log line that prints an arbitrary business identifier inside Body is searchable, but does not thereby populate these correlation fields.
For Python services using the OpenTelemetry zero-code agent, set OTEL_PYTHON_LOG_CORRELATION=true to enable trace context injection into logs. OTEL_PYTHON_LOG_FORMAT controls the printed format; the documented format example includes %(span_id)s. Verify both injection and the collector’s mapping into TraceId and SpanId: printing a span ID in a line is not the same step as normalizing it into the log data model.
Define an application log schema with custom attributes such as request_id, workflow_run_id, attempt, and release_id. These names are application choices, not OpenTelemetry standard fields. Record the mapping between a parent workflow and its individual outbound requests. This lets you distinguish three retries of one dependency from three independent user operations, including when a dependency supplies only logs.
At NIM, NIM_JSONL_LOGGING=true enables JSON Lines output from NIM and its vLLM backend to stderr. NIM_LOG_LEVEL controls verbosity, with WARNING the documented default; VLLM_LOGGING_LEVEL can override backend verbosity independently. Parse and normalize these records in the logging pipeline. JSON output makes parsing predictable; it does not guarantee OpenTelemetry log-field mapping or attach missing context.
Carry both identifiers into inference
On NIM inference endpoints, including /v1/chat/completions, supply X-Request-Id for request correlation and traceparent for causal context. NIM forwards X-Request-Id to the backend as its internal request identifier. It echoes that header only when the caller supplied it; a request without one does not receive a synthesized response header.
Produce NIM inference spans using the backend configuration and endpoint scope taught in 4.3.3. To connect a client span to NIM logs, record the outgoing X-Request-Id value on that inference span as an application-defined attribute, such as request_id; NIM uses the same value in backend log entries.
Test the join in both directions: start with the client span and find the backend log records, then start with a backend error and recover its calling span and parent workflow. Preserve a distinct request ID for each outbound attempt and its mapping to the workflow. A trace that shows one successful inference call must not absorb the log records from a different, failed retry.
Key distinctions and trade-offs
| Identifier or signal | What it establishes | What it cannot establish alone |
|---|---|---|
X-Request-Id | A correlation handle in NIM backend logs | Parent-child relationships across calls |
| Trace ID | Membership in one distributed execution | Which individual operation emitted a record |
| Span ID and parent relationship | The operation and its causal placement | Business-level completion or answer correctness |
| Structured log | A timestamped event with searchable fields | A complete trace when context was never propagated |
More detailed logging increases diagnostic coverage and collection volume. Prefer identifiers, operation outcomes, and bounded metadata over copying request bodies into every event. OpenTelemetry warns against carrying credentials or PII in baggage because propagation can expose it downstream. At an untrusted boundary, define which incoming context you accept and which outgoing context you propagate.
Architecture / decision scenario
An insurer’s employee assistant handles 2,000 requests per minute through an API, a workflow service, a policy tool, and NIM. Its six-second response target is breached intermittently. Operations proposes enabling DEBUG everywhere and searching by timestamp; estimated log volume would rise from 12 GB to 90 GB per day. Twenty concurrent requests can hit the same tool, so that approach still leaves ambiguous joins.
The platform owner instead requires a correlation test before increasing verbosity. The API assigns a workflow identifier, each outbound call records its request identifier, and HTTP instrumentation carries traceparent. Request-processing logs include active trace and span context. The inference adapter sends both NIM headers and records each outgoing request ID on the corresponding client span.
In a controlled batch of 100 workflows, all traces contain inference spans and all requests have backend logs, but only 92 client spans carry the request-ID attribute needed to join them. The platform owner repairs the adapter path that omitted the attribute for eight calls, then repeats the batch. Only after both directions of the join work does the team use the logs to explain a dependency failure in its workflow context.
Common exam traps
- Treating a request ID as a substitute for
traceparentpropagation. - Reusing one span ID for every service in a workflow.
- Assuming JSON Lines automatically contains normalized OpenTelemetry context.
- Assuming a printed span ID is already mapped into structured log context.
- Reusing a request ID across attempts and joining the wrong backend error to a span.
- Expecting NIM to synthesize
X-Request-Idwhen the caller omits it.
What to remember
Propagate context, emit logs with active trace and span identifiers, and retain the mapping between workflow and dependency requests. Verify correlation with a real inference call. Header forwarding, span creation, and log normalization are separate responsibilities with separate failure points.
Knowledge check
An API service puts summarization jobs on a message queue, and a worker process picks each one up and calls NIM. Every job payload carries the request’s request_id, yet the worker’s spans appear as new traces unrelated to the API’s request span. What should the team change?
OTEL_PYTHON_LOG_CORRELATION=true in the worker processrequest_id as an attribute on every worker spanA Python tool service runs under the OpenTelemetry zero-code agent with OTEL_PYTHON_LOG_CORRELATION=true and a log format that prints both the active trace ID and span ID. Both IDs appear in every request-processing log line, but the log backend cannot join those records to spans through their TraceId and SpanId fields. What is missing?
NIM_JSONL_LOGGING=true on the NIM that the tool callsrequest_id value to the Body of each log recordTraceId and SpanId fieldsBody is searchable but does not populate TraceId or SpanId .A workflow’s inference adapter calls a NIM /v1/chat/completions endpoint and retries a failed call up to twice. Engineers must be able to start from any backend error in NIM’s logs and find the client span of the exact attempt that produced it. What should the adapter do?
X-Request-Id on every attemptX-Request-Id per attempt and record it on that attempt’s spanX-Request-Id and record the identifier that NIM’s response returnstraceparent and match the backend logs by their timestampsX-Request-Id as the backend request ID in its logs, so a distinct value per attempt, recorded on the matching client span, joins each backend record to one attempt. A shared value merges the attempts, NIM synthesizes no header when the caller omits one, and timestamps are ambiguous under concurrent traffic.A tool gateway receives traceparent from the workflow service. To make joins easy, its developer writes the incoming parent span ID into every record that the gateway and its downstream calls emit. What should the gateway do instead?
request_idReferences
- OpenTelemetry, Context propagation
- OpenTelemetry, Logs data model
- OpenTelemetry, Python zero-code instrumentation configuration
- NVIDIA, Observe Workflows
- NVIDIA, Logging and Observability