Tool Evaluation

Why it matters

A redesigned tool set is only better if measurement says so. 3.1 asks for evidence that a narrower or renamed catalog still completes the work, and this harness is a minimal way to collect it.

What to practise

Load prompt-and-answer tasks from an XML evaluation file and let an agent solve each one with the tools you pass. The harness scores exact-match answers, times every call, and counts calls per tool. The evaluator prompt also makes the agent return <feedback> on tool names, parameters and descriptions. Run it on the supplied calculator tool, whose description fields are deliberately empty, and read the report. The notebook stops there: fill the descriptions in yourself and rerun, watching accuracy, tool calls per task, and whether the feedback changes.

How it fits

This is 3.1.4’s validation path at small scale. Extend the task file with the selection, denial and recovery cases 3.1.4 names, and use the per-tool counts to find the selection errors that 3.1.1’s inventory looks for.