Standalone ATIF Evaluation with Custom Evaluators

Why it matters

A correct answer says nothing about how the agent reached it. Scoring the trajectory beside the answer is what tells you which decision to fix.

What to practise

Write two trajectories as ATIFTrajectory objects — one that calls a calculator tool and answers correctly, one that answers wrongly without tools — and pair each with its expected and actual output in an AtifEvalSample. Implement two evaluators on AtifBaseEvaluator, each overriding only evaluate_atif_item: an exact match on the answer and a count of the tool calls in the trajectory’s steps. Run them through EvaluationHarness.evaluate() and read each item’s score and reasoning. Note how little the scoring needs: nvidia-nat-eval alone, with no LLM endpoint, YAML config, toolkit workflow or evaluator plugin.

How it fits

The two evaluators are 2.6.1’s separation in its smallest form: an answer score and a trajectory score kept apart. Score candidate strategies on the same samples, as 2.6.2 requires, before deciding what to change.