Prove the sample before scaling submissions

Learning objective

Test and refine on a sample before large batches, accounting for oversized documents, polling, and reruns.

Core idea

A large batch amplifies the request you actually submit, including its defects. If the prompt produces plausible but unusable findings on a sample, batching more inputs does not repair it. It creates more outputs to inspect and potentially regenerate.

Separate two preflight questions: does the request work, and does its answer satisfy the task? Anthropic recommends dry-running a request shape through the Messages API before batching. Use representative input samples to refine quality as well, then run a small batch to verify the deferred lifecycle your production consumer must handle.

See 4.5.4 for the result-recovery algorithm. This lesson focuses on preventing repeated full-volume failures and making a measured decision to scale.

How it works

Test request validity and answer usefulness separately

The batch guide states that validation of each entry’s params happens asynchronously, with errors available after batch processing ends. A successful submission response therefore does not prove that the individual message requests are valid. Discovering a shared defect only after thousands of entries reach terminal results wastes a processing cycle.

Dry-run the message shape using POST /v1/messages. Test the actual instructions, document representation, and output requirements intended for the batch. When assembling the batch, place the supported message parameters inside each entry’s params. A valid synchronous request still needs adaptation to batch restrictions: stream: true, speed, and max_tokens: 0 are not supported in batch requests.

Then evaluate the answer against the job’s purpose. For test-gap analysis, require a finding to identify the untested behavior and connect it to supplied source and tests. For extraction, require the values to match the source evidence. Those acceptance criteria are application decisions; result.type: "succeeded" only reports successful message creation.

Choose a sample that exercises the variation you expect to submit. Include short inputs, large inputs, sparse evidence, and sources with no qualifying findings.

Refine the prompt on observed failures, then evaluate the revision on separate representative examples before scaling.

Preflight the document and the batch independently

The token-counting endpoint, POST /v1/messages/count_tokens, returns an estimated input_tokens value for the supplied structured input. For supported inputs, compare that estimate with your chosen model’s context window to identify documents that need a different submission shape before spending a batch cycle on them. Include instructions and other context in the count rather than measuring the document’s character count alone.

Token counting supports client tools but rejects most server tools and the MCP connector. It also rejects image or document blocks with url or file sources; send images and PDFs as base64 when counting them. For server-tool requests that the endpoint cannot count, the Messages response’s usage provides actual usage evidence. Do not describe token counting as a universal preflight for every request accepted by Messages.

The returned count is an estimate, so leave room rather than treating it as an exact admission certificate. Input alone exceeding the context window produces HTTP 400 invalid_request_error with “prompt is too long” on every model; in batch processing, inspect the errored entry as described in 4.5.4.

Input that fits can still leave insufficient room for generation. On Claude 4.5 models and later models, the API accepts a request when input tokens plus max_tokens exceed the context window. If generation reaches that window, the message stops with stop_reason: "model_context_window_exceeded". Earlier models reject that combination with a validation error by default. Thus context pressure can appear as an errored entry or as a succeeded entry whose message stopped at the context limit. Check the message’s stop reason as part of sample acceptance, and leave room for the output your task needs.

If the error explanation or preflight shows that a document is oversized for the chosen request, split it into coherent parts and test whether the parts still support the required analysis. Assign each part a separate custom_id and retain the source mapping described in 4.5.4.

Chunking changes the evidence available to each request. A review that needs relationships across files may require a separate integration analysis after the parts return. Do not claim that reducing input size alone preserves answer quality. The sample must test the transformed inputs, not only the original prompt on small complete documents.

Batch size is a separate boundary. The documented limit is 100,000 message requests or 256 MB, whichever is reached first. An oversized batch request can return HTTP 413 request_too_large. Splitting a large submission into multiple batches addresses that envelope limit; it does not shorten an individual document that exceeds its own input budget. Conversely, chunking one document does not establish that the aggregate serialized batch fits.

Exercise the asynchronous consumer before full volume

After synchronous preflight, use a small batch to exercise the actual batch lifecycle. Save its id, inspect it through GET /v1/messages/batches/{message_batch_id}, and wait for processing_status: "ended" before retrieving results. Observe request_counts so the test includes terminal categories, not only the happy-path file download.

Choose a polling interval deliberately. Frequent polling reduces the delay before your application notices completion; it does not make inference finish sooner. Include that detection delay in delivery planning, as described in 4.5.3. A sample that validates only the prompt leaves the report consumer untested.

Verify that the consumer matches results by custom_id even when order differs. Exercise its handling of a validation failure and an expired outcome without assuming either outcome will occur naturally in every sample batch. The result reference defines the envelopes your consumer needs to accept; you do not need to wait for a real 24-hour expiration merely to inspect that contract.

Price reruns from accepted output

Measure first-pass acceptance: how much submitted work produces usable output without another generation. A technically successful message can still fail your quality criteria, so report that separately from API error counts.

For an illustrative run of 1,000 requests, a prompt defect requiring 100 successful answers to be regenerated creates 100 additional generations. The batch discount applies to the usage; it does not erase the first generation’s cost. If a sample exposes that defect, repairing the prompt before the large run avoids amplifying it.

Keep accepted results when only a subset needs recovery. When you intentionally change the output contract for every source, a broader rerun may be justified, but that is a separate quality decision. Do not call it transport recovery or silently discard useful outputs because some neighboring entries failed.

Key distinctions and trade-offs

Synchronous preflight catches request-shape problems quickly; sample evaluation checks answer usefulness; a small batch checks deferred result handling. Each answers a different question. Passing one does not prove the others.

Observed failures and an oversized document both return to sample evaluation before the batch scales.

Document token size and total batch byte size also require different fixes. Use token counting where supported and test the document transformation. Use the batch’s count and byte limits to partition the submission. Neither check replaces evaluating the actual output.

Choose sample coverage from workload variation and failure impact rather than an assumed universal sample size.

Architecture / decision scenario

A team plans nightly test generation over thousands of modules. It first tests the intended Messages request on representative modules, including a large legacy module and a module whose behavior is already covered. The sample reveals that the prompt invents gaps when none exist and that the legacy input is oversized.

The team revises the instructions to permit no findings, checks that behavior on separate examples, and preflights the large input with count_tokens. It splits the legacy module, gives the parts distinct identifiers, and evaluates whether the resulting findings retain the needed context.

A small batch then tests polling, unordered result joins, and selective recovery. The full dataset is partitioned against both the 100,000-request and 256 MB limits. The team scales only after it can distinguish API success from accepted test proposals and can rerun the required subset without regenerating accepted outputs.

Common exam traps

• Treating batch acceptance as validation of every entry’s params.
• Scaling after checking only easy inputs or a single attractive answer.
• Confusing the total batch byte limit with a document’s token budget.
• Assuming token counting accepts every server tool and document source representation.
• Expecting faster polling to speed up batch processing.
• Ignoring the cost of regenerating technically successful but unusable answers.

What to remember

Prove the request shape, the answer quality, and the asynchronous consumer before scaling. Test transformed oversized inputs, partition against both batch limits, and measure usable first-pass output. Prompt refinement and selective reruns preserve the batch discount’s value by reducing repeated generation.

Knowledge check

References

• Anthropic, Batch processing

• Anthropic, Token counting

• Anthropic, Retrieve Message Batch results

• Anthropic, Context windows