Simple LangSmith-Documentation Agent - Evaluation and Profiling

Why it matters

Comparing models is only meaningful when everything but the model stays fixed. This example keeps the agent, dataset and evaluators, and its alternative configuration files show how easily more than the agent’s model changes.

What to practise

Run nat eval on the simple_web_query agent — a react_agent that answers questions about the LangSmith documentation through a webpage_query tool — once per configuration: eval_config.yml for NVIDIA Nemotron, then the OpenAI, Llama 3.1 and Llama 3.3 variants. Diff each file against the default before reading its scores: the OpenAI variant moves the judge LLMs to the same OpenAI model as the agent, and the Llama 3.1 file changes the judge models and drops profiler options. Compare the per-evaluator files such as accuracy_output.json and trajectory_accuracy_output.json, and the profiler’s standardized_data_all.csv, workflow_profiling_report.txt and gantt_chart.png. Rerun the evaluators alone with --skip_workflow over a saved workflow_output.json.

How it fits

The configuration diff is 3.2.1’s control check: a matched comparison holds the dataset and the judges fixed and changes only the component under test. Read the accuracy files beside the profiler’s latency and token data to make 3.2.3’s choice of quality against cost and latency.