AnthropicClaude Certified Architect – Professional2026-102.3 Apply prompt engineering techniques (zero-shot, few-shot, chain-of-thought)
Justify prompting complexity with measured gains
Completed by the knowledge check
Learning objective
Choose among zero-shot, few-shot, and reasoning techniques by measured quality gain versus token and latency cost.
Core idea
For an architect funding a production Claude workflow, the winning technique is the least expensive acceptable configuration for the decision. A quality improvement has value only in relation to the mistakes it prevents, the response time the user can tolerate, and the traffic that pays for it.
Zero-shot, few-shot, and reasoning are candidates to compare, not mandatory steps in a ladder. Examples may resolve a local convention without requiring deeper reasoning. Thinking may improve a multistep decision without additional examples. A combined candidate deserves deployment only when its measured benefit pays for the additional work. See 2.3.1–2.3.3 for how to construct each candidate; this lesson owns the comparison.
How it works
State the acceptance rule before running candidates
Anthropic’s evaluation guidance calls for measurable, relevant success criteria and multidimensional evaluation. Translate the business requirement into a release rule: required decision quality, acceptable consequential error rates, a latency target, and a cost ceiling at forecast traffic.
Use the same held-out cases and grading rules across candidates. Preserve the real task distribution and examine edge cases separately. For a classification task, exact-match grading can measure the label; a policy judgment may also need a rubric for whether the cited rule supports it. Automate what you can verify mechanically and retain domain review for the judgments the rubric cannot settle reliably.
Define how refusals, incomplete results, and unresolved cases affect the score. Dropping them from the denominator can make a slow or unreliable candidate appear accurate. Report accepted outcomes and terminal failures separately, with the same counting rule for every configuration.
Change the technique without changing the experiment
Freeze the model, input context, result requirements, and grading policy. Compare the zero-shot baseline, an example-only change, a reasoning change, and a combined configuration when the failure analysis justifies both. Record each prompt version, example-set version, thinking configuration, output_config.effort, and max_tokens.
“No examples” and “no thinking” are different conditions. If the selected model cannot disable thinking, label the baseline zero-shot at a specified effort level and compare reasoning intensity rather than claiming an off-mode experiment. Do not change the model at the same time as the technique and attribute the result entirely to prompting. Model-tier comparisons belong in 2.1.1.
The experiment should isolate the question it answers. If examples fix convention errors while more reasoning fixes exception interactions, test those slices independently before applying the combined prompt to every request. One aggregate score can conceal that a technique helps only a small part of traffic.
Count inputs, then measure actual output work
On the Claude API, POST /v1/messages/count_tokens, or Python’s client.messages.count_tokens(...), estimates input tokens for the target model and request. Use it to compare the added size of candidate example sets before inference. The count is an estimate, and support is narrower than message creation: for example, most server tools and the MCP connector are rejected by the counting endpoint. For those requests, use actual response usage.
After inference, record actual token usage rather than inferring cost from visible text. usage.output_tokens is the inclusive output total used for billing. usage.output_tokens_details.thinking_tokens breaks out the reasoning portion; it is already included in output_tokens, so adding it again double-counts the spend. With streaming, that breakdown arrives on the final message_delta event.
Changing thinking display does not make inference cheaper: summarized and omitted display bill the same internal work. A short visible answer can therefore conceal substantial output-token consumption. Use measured totals and the deployment’s applicable rates, rather than multiplying the length of a displayed summary by a price.
Count all attempts needed for an accepted result. An inexpensive first response that frequently needs repair can cost more than a stronger candidate that finishes once. See 1.6.2 for the full workflow cost ledger; the technique comparison feeds its measured attempts and token usage into that ledger rather than replacing it.
Measure latency separately from tokens
Record end-to-end response time from request start to an accepted result, including repairs or decomposed stages. Measure the latency percentile required by the product, not only the average. If the interface streams progress, time to the first visible text and time to an accepted decision are different measures; early text does not satisfy a deadline for a usable result.
Native thinking spends time before the answer, while a chain of calls introduces stage boundaries and repeated processing. Token reduction is a useful cost signal, but it does not by itself prove that the user-facing latency target is met. Measure the complete candidate under comparable conditions.
For supported models, output_config.effort is a calibrated control for response work. Top-level effort is available on all supported models with no beta header required; per-message effort is a separate beta feature. This comparison uses the top-level setting. Lowering it is a candidate change to evaluate, not a promise of unchanged quality or a fixed token budget.
Choose by incremental value and traffic
First eliminate candidates that fail a required quality or latency boundary. Among the acceptable candidates, compare the extra cost per avoided consequential mistake and the total expense at forecast volume. Preserve the evaluation results in the prompt-release decision so reviewers can inspect the reason for adopting complexity.
A route used once in a hundred requests can justify a richer prompt that would be wasteful on every request. However, a selective route needs a measured selection rule: if the routing signal misses the difficult cases, their quality requirement still fails. Evaluate the complete routed workflow, including selection mistakes, before using slice-level results to claim production quality.
Key distinctions and trade-offs
The techniques buy different kinds of work:
| Candidate | What must improve | Recurring cost to measure |
|---|---|---|
| Zero-shot | Decisions from explicit instructions and context | Baseline input and output usage |
| Few-shot | Transfer of conventions and boundary judgments | Demonstration input tokens and maintenance |
| Reasoning | Decisions requiring intermediate work | Thinking output tokens, latency, and any extra stages |
| Combined | Both error classes enough to justify both additions | Total inputs, outputs, attempts, and elapsed time |
A gain in low-impact labels can be less valuable than a smaller gain in a costly exception category. The business owner approves that value judgment; the architect supplies the comparable measurements.
Architecture / decision scenario
A retailer classifies 100,000 product listings daily, using separate calls for category and restriction code. The business requires 97% category accuracy and 96% restriction-code accuracy, with a 2-second p95 deadline for the ordinary route. All figures here are illustrative measurements for one fixed deployment.
On the category slice, zero-shot reaches 97.8% at 0.8 seconds. Four examples reach 98.0% at 1.0 seconds but add 300 input tokens per request. The architect retains zero-shot: the required quality is met, and the extra examples add 30 million input tokens daily for a marginal gain the business owner declines to fund.
Restriction codes depend on local conventions. Zero-shot reaches 82%, examples reach 97% at 1.2 seconds, and an increased-effort candidate without examples reaches 87% at 2.8 seconds. The restriction-code examples add 400 input tokens per call, or 40 million daily across 100,000 listings. That cost buys the improvement from 82% to 97% needed to meet the 96% bar within the latency target; extra reasoning fails both requirements. The business owner approves the expense, and the architect selects the example set for this separate call.
A separate exception queue handles 1,000 listings daily with a 10-second review target. Examples alone achieve 88% agreement, and examples plus higher effort achieve 95% against a 94% requirement at 6 seconds. Measured thinking adds 1,500 output tokens per case: 1.5 million daily on that queue, compared with 150 million if applied to all traffic.
The architect selects the combined technique only for the exception route after evaluating its selection rule. The release record contains held-out scores, actual usage, end-to-end latency, and the business owner’s approval of the measured expense. Each decision gets a justified technique rather than inheriting the most complex prompt.
Common exam traps
- Selecting the highest aggregate score without checking consequential errors and latency.
- Changing the model alongside examples and claiming the gain came from few-shot prompting.
- Adding
thinking_tokenstooutput_tokenseven though it is already included. - Treating omitted thinking display as a cost reduction.
- Evaluating a difficult slice while ignoring mistakes in the route that selects it.
What to remember
Compare controlled candidates on held-out quality, actual token usage, and latency to an accepted result. Pay for examples where conventions improve, and reasoning where intermediate work improves decisions. Fund a combination only where the measured benefit justifies its recurring cost and the whole routed workflow meets requirements.
Knowledge check
A cost dashboard for a Claude API workflow computes each request’s output spend as usage.output_tokens plus usage.output_tokens_details.thinking_tokens. What is wrong with that figure?
usage.output_tokens is the inclusive billed output total, and thinking_tokens breaks out the reasoning portion already inside it, so adding the two double-counts reasoning. With streaming the breakdown arrives on the final message_delta event, but it is the same figure, not an extra charge.Finance asks a team to cut thinking spend on a Claude Opus 5.5 workflow without dropping below its held-out quality bar. Which change can actually reduce the billed thinking work?
output_config.effort and rerun the held-out evaluationdisplay to "omitted" on every requestA retailer classifies 100,000 product listings a day into categories. The release rule requires 97% accuracy within a 2-second p95 at the lowest recurring cost. Zero-shot reaches 97.8% at 0.8 seconds; four examples reach 98.0% at 1.0 seconds and add 300 input tokens per request. Both candidates have the same measured output usage and need no retries; higher-effort variants have not been evaluated. Which configuration should the architect select?
On a 1,000-listing-a-day exception queue, examples plus higher effort reach 95% agreement against a 94% requirement. The team plans to apply that configuration only to listings a routing rule sends to the queue. What must be evaluated before claiming the exception requirement is met in production?
count_tokensReferences
- Anthropic, Define success criteria and build evaluations
- Anthropic, Token counting
- Anthropic, Steering thinking
- Anthropic, Effort