Establish a reproducible zero-shot starting point

Learning objective

Establish a zero-shot baseline with clear, specific, well-structured instructions.

Core idea

For an architect taking a Claude workflow into production, explicit task criteria — what qualifies and what does not — are the starting point. This lesson adds the decision that follows: make a clear prompt the measured baseline against which every additional prompting technique must earn its cost.

Zero-shot means that you supply no worked input-output demonstrations. It does not mean that you omit business rules, source material, output requirements, or context explaining the task. Nor does it mean that model-native thinking is disabled. Record the thinking configuration separately; otherwise a supposedly simple baseline can already include substantial reasoning work.

A baseline is useful only when another configuration can be compared with it. Give the prompt a version, define the task’s success criteria, and evaluate it on cases the prompt has not answered in advance. Anthropic’s evaluation guidance recommends task-specific tests reflecting the real distribution and edge cases, with measurable quality and operational criteria.

How it works

Define the task before choosing a technique

Write down the input the workflow actually receives, the decision it must return, and the business consequence of a wrong answer. Anthropic’s prompting guidance recommends clear, explicit instructions, specific output constraints, and context explaining why the behavior matters. Translate those recommendations into a prompt that a colleague with minimal background can apply consistently.

For an education provider assigning incoming service requests, “categorize the message” leaves the categories, tie-breaker, and missing-information behavior undefined. The zero-shot prompt needs definitions of the permitted categories, a rule for messages containing two requests, and a way to represent insufficient information. Those are instructions, not examples.

Do not confuse a rule with a demonstration. “Choose the billing category when the request concerns an invoice” defines a criterion. A complete sample message paired with its correct classification demonstrates that criterion and changes the prompt into few-shot. Keep the distinction explicit in the experiment record so you know which intervention produced a gain.

Make structure visible without inventing an interface

Anthropic recommends descriptive XML tags such as <instructions>, <context>, and <input> to distinguish mixed prompt content. Use consistent tags to separate the task rules, supplied facts, and live request. These tags organize text; their names are not Messages API fields and do not validate an answer.

For a short classification prompt, keep the category definitions together, put the tie-breaker beside them, and distinguish the student’s message from the instructions. When order or completeness matters, use numbered steps to specify the required work. Do not add a ritual sequence to a direct lookup merely to make the prompt appear more rigorous.

The request’s system field can hold durable instructions, while messages carries the live input. The baseline record should identify both contents rather than storing only the user-facing question. Changes to either can change the experiment. Structured prompts remain zero-shot when they contain no demonstrations, regardless of how detailed the definitions become.

Freeze the comparison conditions

Use an evaluation record containing the prompt version, model, thinking configuration, output_config.effort where supported, max_tokens, supplied context version, and grading rules. See 2.1.3 for selecting supported reasoning controls. Here the purpose of recording them is to prevent a change in model behavior from masquerading as better instructions.

Set quality thresholds before looking at candidate results. A routing task might need high recall for account-access problems and a limit on unnecessary escalation, rather than one aggregate accuracy score. Measure those requirements independently. Include incomplete, ambiguous, and mixed-intent inputs because ordinary successes do not establish that the prompt’s fallback rules work.

Keep the test cases out of the prompt. If an evaluation case becomes a demonstration during iteration, move it into the development set and replace it in the held-out set. Otherwise the score starts measuring reproduction of supplied answers. Anthropic’s guidance on held-out tests and comparison with a baseline supports treating evaluation as a separate artifact.

Diagnose what the baseline cannot do

Inspect failed cases before selecting the next technique. A missing category definition calls for clearer instructions. A policy clause absent from the request calls for the relevant context. A house convention that remains ambiguous after definitions may justify examples; see 2.3.2. A conflict requiring multiple intermediate decisions may justify reasoning; see 2.3.3.

This classification prevents an expensive intervention from concealing the actual omission. A larger example set cannot establish the correct version of a policy you never supplied. More thinking cannot resolve two contradictory category definitions. Repair the information and instruction defect, rerun the baseline, and only then compare a technique change.

Key distinctions and trade-offs

Zero-shot versus underspecified: A zero-shot prompt can contain extensive domain context and precise constraints. Omitting demonstrations saves their input tokens and curation work; omitting necessary rules makes the task undefined.

Prompt technique versus thinking mode: The absence of examples says nothing about whether the selected model thinks. On Claude Opus 5.5, adaptive thinking is on by default and cannot be disabled. A baseline using that model must record this behavior rather than label itself “no reasoning.”

Baseline versus production approval: A reproducible starting point establishes a comparison, not permission to deploy. It may fail the quality requirement. The architect owns the comparison design; the business owner owns whether the measured errors are acceptable.

An instruction-only prompt is easier to maintain when written definitions settle the task. It becomes a poor production choice when unseen cases show that those definitions leave important distinctions unresolved. The measured result decides whether simplicity is sufficient.

Architecture / decision scenario

An education provider routes 25,000 support messages daily. Its service owner requires at least 98% recall for account-access requests and fewer than 5% unnecessary escalations. The team has 1,200 held-out messages, including missing student identifiers and messages combining payment and access problems. These are illustrative requirements and measurements.

One proposal adds ten historical messages immediately. The architect instead establishes a zero-shot candidate with category definitions, a stated priority for access problems, and an insufficient-information outcome. <instructions>, <context>, and <input> separate the prompt parts. The team freezes model, thinking settings, and grading rules for the comparison.

The first run reaches 94% access recall. Failure inspection shows that temporary lockouts were never included in the access definition. The architect repairs the definition rather than adding demonstrations of ten lockouts. The revised baseline reaches 98.4% recall with 3.2% unnecessary escalations and a 1.1-second p95 response time against a 1.5-second target.

The service owner accepts those error rates. The architect retains the zero-shot configuration as the production candidate and the comparison point for later experiments. If a subsequent few-shot candidate improves ordinary-category accuracy but misses the access recall requirement, it does not replace the baseline. Clear instructions and measured requirements jointly determine the choice.

Common exam traps

• Equating zero-shot with a short prompt containing no business context.
• Assuming that no examples also means no model-native thinking.
• Adding demonstrations before fixing an omitted definition or missing source.
• Calling a run a baseline while changing the model and grading rubric between candidates.
• Using aggregate accuracy to hide a failure on the business-critical category.

What to remember

Establish zero-shot as a reproducible configuration: clear task, explicit constraints, separated context and input, and no worked demonstrations. Record the model and thinking settings independently. Diagnose failures before paying for examples or reasoning, and retain the simplest candidate that meets the actual requirements.

Knowledge check

References

• Anthropic, Prompting best practices
• Anthropic, Define success criteria and build evaluations
• Anthropic, Thinking