Define acceptance before designing the solution

Learning objective

Translate business goals into measurable solution capabilities and acceptance criteria, including tolerable error, data constraints, human roles, and system boundaries.

Core idea

A capability states what the solution produces. An acceptance criterion states the evidence required to accept that output or the solution as a whole. “Summarize referral documents” is a capability; “reduce preparation time without omitting the facts clinicians need” requires a measurement, a rubric, and an accountable reviewer.

Anthropic defines good success criteria as specific, measurable, achievable, and relevant. It also calls for multidimensional evaluation: task fidelity alone does not settle consistency, privacy preservation, latency, or price. Your architecture begins with an acceptance contract that makes these demands compatible and exposes any conflict before implementation.

This Bite defines the contract. Detailed metric selection and evaluation execution belong to 4.1 and 4.2; the proof-of-value experiment belongs to 1.1.4. Here, decide whose requirements constrain the system and what evidence would satisfy them.

How it works

Convert the goal into observable capabilities

Break a goal such as “reduce preparation time” into outputs a person or downstream service can inspect. Name the input population, output, recipient, and intended use. A document summary for a clinician and extracted administrative fields for a scheduling service are separate capabilities, even when one model call produces both.

For each capability, record a baseline and a candidate target. Use prior experiments or expert judgment to justify the target, as Anthropic’s success-criteria guidance requires. Label provisional targets as hypotheses. A sponsor’s desired improvement is useful input, but it is not evidence that the model can achieve it.

State the denominator. “98% correct” can mean fields, documents, accepted outputs, or all incoming cases. A system that rejects half the workload can perform well on accepted outputs while creating no operational benefit. Record both quality and the share of cases that require human handling.

Specify tolerable error by consequence

Distinguish omitted evidence, invented evidence, an incorrect field, and an unusable response. Give each a rubric or deterministic check, a tolerable rate, and a consequence. Anthropic explicitly includes edge-case handling in task fidelity and offers both quantitative metrics and consistently applied expert scales.

An overall average can conceal the error that matters most. A missed contact phone number and an invented medication instruction should not share one undifferentiated correctness score. A critical-error gate can fail even when most administrative fields are accurate. “No critical errors observed” describes a test result; it does not establish zero production risk.

Name the artifact that enforces the decision: an acceptance matrix with the error definition, evaluation population, threshold, reviewer, and failed-gate disposition. For a semantic defect, the matrix can require a domain expert’s documented review. For a malformed field, it can require an application validation check before any write.

Connect the output contract to the API boundary

When a Claude API capability returns structured fields, JSON outputs use output_config.format with type: "json_schema". This is a structural mechanism, not a guarantee that extracted facts are true. Structured outputs are generally available on supported models on the Claude API, Claude Platform on AWS, Google Cloud, and Microsoft Foundry. On Amazon Bedrock, they are available only through the legacy integration for Claude Opus 4.6, Sonnet 4.6, Sonnet 4.5, Opus 4.5, and Haiku 4.5, not through Claude in Amazon Bedrock.

The acceptance contract also covers exceptional responses. A safety refusal has stop_reason: "refusal", can return HTTP 200, and can violate the requested schema because the refusal takes precedence. Do not count HTTP success as a completed capability. Record a non-completed outcome and retain the human workflow. Generated refusal tokens are billed, so count refused calls in the contract’s processing cost. A schema property requesting the model’s thinking or step-by-step reasoning can cause a reasoning_extraction refusal; ask for a short explanation instead.

A response ending with stop_reason: "max_tokens" can be incomplete; the documented remedy is to retry with a higher max_tokens. If that retry exceeds your service deadline, keep the case in the human workflow rather than passing partial fields onward. String enum and const values also have a documented casing exception: compare case-insensitively and avoid values distinguished only by capitalization. Business acceptance must handle these exceptions instead of relying on the schema label alone.

A safety refusal, or an incomplete response whose retry exceeds the service deadline, stays in the human workflow.

Make data and human roles part of acceptance

Define the allowed input sources and fields, who authorizes their use, and what evidence must remain available to reviewers. A data constraint is a precondition for running the capability, not a preference the model can decide to ignore. In the contract, identify the integration or preprocessing stage responsible for excluding disallowed data.

A privacy-preservation evaluation tests how outputs handle sensitive information, as Anthropic’s guidance describes. It does not prove that a particular deployment, retention configuration, or contractual arrangement satisfies the organization’s obligations. Compliance and legal owners determine those requirements; your data owner approves the input scope before the experiment.

Give the human a concrete job. “Human in the loop” does not tell you who checks omissions, how quickly they must respond, or who acts when they disagree. Specify reviewer qualifications, evidence shown, approval authority, and the system’s behavior before approval. Include correction and review time in the business outcome so the model cannot appear beneficial by shifting work to another queue.

Key distinctions and trade-offs

The acceptance contract needs several independently enforced gates:

Higher quality can require more review, which consumes the time savings. Narrowing inputs can protect data boundaries while lowering coverage. Decide which constraints are hard gates and which targets can be traded, then preserve that distinction in the contract.

Architecture / decision scenario

A healthcare referral office with four coordinators wants preparation time to fall from ten to six minutes per eligible case. The capability extracts administrative fields and drafts a document summary; diagnosis and treatment decisions remain outside the system. All numbers are illustrative organizational targets.

The data owner authorizes a defined document set and requires unrelated attachments to be removed before requests. Legal and compliance owners must approve the deployment’s handling of those records. The service does not begin a production pilot until those preconditions hold.

The acceptance matrix requires at least 98% administrative-field correctness on the labeled sample, separate reporting for missing-document cases, and no observed invented clinical facts. An invented fact fails the gate regardless of the average. A clinician reviews the summary against the source before it is used, while a coordinator confirms administrative fields before the scheduling service records them.

The integration uses JSON outputs only on a verified supported model and platform. It treats refusals and truncated output as incomplete cases and preserves manual preparation. The candidate must meet the six-minute target including both reviewers’ time, and the office requires 95% of eligible requests to complete preparation within thirty minutes. These system boundaries, reviewer duties, data preconditions, and failure dispositions define acceptance more precisely than a single accuracy score.

Common exam traps

• Treating a business aspiration as an evidenced acceptance threshold.
• Reporting accuracy without the population, denominator, or error severity.
• Counting HTTP 200 or valid JSON as proof of a correct completed task.
• Using privacy evaluation as evidence of legal or contractual compliance.
• Naming a human reviewer without assigning authority, workload, and timing.
• Excluding correction time or incomplete cases from the claimed business benefit.

What to remember

Write an acceptance contract that ties capabilities to evidence, thresholds, and owners. Include severe errors, data preconditions, human responsibilities, and system boundaries. Schema control supports one gate; it does not replace semantic review or business acceptance.

Knowledge check

References

• Anthropic, Define success criteria and build evaluations
• Anthropic, Structured outputs