RAG Agent Evaluation

Why it matters

A score is only as good as its dataset, its judge and its denominator. This notebook makes you handle all three on a RAG agent.

What to practise

Load a fixed JSON dataset of questions, reference answers and categories, run the IT help desk agent over it, and score each answer with a rubric judge for faithfulness, relevance and helpfulness. Then compute the four RAGAS metrics — context precision, context recall, faithfulness and answer relevancy — and add two custom checks: whether every [KB:source_id] citation resolves to a retrieved chunk, and an actionability rubric. Watch failed measurements stay null rather than low, record your own ratings beside the judge’s, and save the run so compare_runs can pair it with a later one on the same dataset hash.

How it fits

The dataset is 3.1.1’s benchmark set, the human review starts 3.1.2’s calibration, which the notebook says three examples cannot finish, and the RAGAS split is 3.1.3’s separation of retrieval from answer failures. The saved run directory is the small-scale version of 3.1.4’s repeatable run.