Code based grading

Why it matters

A model grader can judge whether an answer follows the task. Whether generated code parses is a fact that code can check, cheaply and the same way on every run.

What to practise

Write validators that try to parse the output as JSON, Python or a regular expression and score 10 or 0. Add a format field to each test case so the grader knows which validator to apply, and tighten the prompt so that Claude returns only code. The lesson also prefills the assistant turn with a code fence, which Claude 4.6 models reject with a 400 error; on them, the prompt instructions do that job alone. Then average the syntax score with the model grader’s task-following score to get a baseline you can improve against.

How it fits

This is the deterministic-check half of 4.2.2. Weigh the average against 4.2.2’s rule that one score must not hide a critical failure: code that does not parse still earns half marks, so decide whether invalid syntax fails the case outright. The format field is test-case metadata that 4.2.4 keeps correct as the set changes.