Tracing, Evaluating, and Profiling your Agent

Why it matters

A failed case tells you that something went wrong; a trace and a profile tell you where. This notebook puts all three views on one agent so you can join them.

What to practise

Build a tool-calling retail sales agent, then add a general.telemetry.tracing exporter with _type: phoenix and inspect its traces in Phoenix. Add an eval section with ragas evaluators for AnswerAccuracy, ResponseGroundedness and ContextRelevance plus a trajectory evaluator, and run nat eval. Then enable the profiler with compute_llm_metrics, bottleneck_analysis with enable_nested_stack, prompt_caching_prefixes and concurrency_spike_analysis, run nat eval again, and read standardized_data_all.csv, inference_optimization.json and the Gantt chart.

How it fits

Joining low-scoring entries to their traces is 3.5.1’s failure attribution, and the per-call token counts and timeline are 3.5.2’s bottleneck profile. Rerun the same evaluation after a fix to confirm it, as 3.5.3 requires.