Retrieval Augmented Generation

Why it matters

A RAG answer can fail because the right passage was never retrieved or because the model misused it. 3.5 asks you to tell these apart before tuning anything; this notebook measures both.

What to practise

Chunk the documentation by heading, embed each chunk with Voyage AI, and retrieve into an in-memory vector store. Score retrieval with precision, recall, F1 and mean reciprocal rank against each question’s expected chunks, and score answers separately with a Claude judge for end-to-end accuracy. Then add a summary per chunk and a Claude reranking step, and watch which metric moves: mean reciprocal rank can rise far more than precision. The notebook’s prose names earlier models than its code uses; the method is unchanged.

How it fits

The two scorecards are 3.5.5’s split between passage selection and answer quality, and the heading-based chunks are a 3.5.3 boundary decision to question. For 4.4, the same split is 4.4.3’s diagnosis: when end-to-end accuracy drops, check retrieval metrics first to separate a missing passage from an unfaithful answer.