Report Agent Evaluation

Why it matters

A report agent that searches the live web never sees the same evidence twice. Comparing two versions of it takes discipline about what you hold fixed and what you record.

What to practise

Load the report test cases — a topic, expected sections and content criteria for each — and generate a report per case with the Module 1 agent and its Tavily search tool. The run records the agent and judge models, temperatures, prompt and a fingerprint of the rubric. Score each report on structure, content, coverage, accuracy and writing, where accuracy means support against the saved search excerpts, not external fact-checking. Read each mean with its count of successful measurements, then change one component, rerun the same dataset and use compare_runs.

How it fits

The recorded configuration and fixed dataset are 3.2.1’s comparison controls. Live search is a source of run-to-run variation, so read score changes the way 3.2.2 separates noise from improvement, against the evidence each run saved.