Match endpoint bytes to container contracts

Learning objective

Prepare serialized inputs and responses for SageMaker AI inference endpoints.

Core idea

An inference endpoint is a transport boundary around a deployed model and its serving code. Amazon SageMaker AI forwards your request body to the model container. It does not turn an arbitrary application object into the exact input that the algorithm or container expects.

Define both directions of the contract: how your application serializes input, how the container deserializes it, how the container serializes its result, and how your application deserializes that result. AWS identifies these four conversions in its inference data-format documentation.

The contract includes more than a MIME type. It includes field names, feature order, units, record boundaries, required generation settings, output structure, and handling of incomplete or invalid results. Hosting choices are covered in 1.1.3; this lesson concerns the data that crosses the inference boundary.

How it works

Bind the serializer to the deployed contract

For synchronous invocation, the SageMaker Runtime InvokeEndpoint operation takes EndpointName, Body, and headers represented by parameters such as ContentType and Accept. Body is the input data. ContentType declares its MIME type; Accept declares the desired response MIME type.

AWS documents text/csv, application/json, and application/x-recordio-protobuf among inference input formats for SageMaker AI algorithms. Format support differs by algorithm. These examples are not a universal schema for a foundation-model serving container. Consult the deployed algorithm or container contract before choosing a format or JSON structure.

For a CSV model, feature order matters and must remain unchanged. Apply the transformations used for training to inference data as well. A row can parse correctly while assigning every value to the wrong feature. Validate the feature list and transformation version before producing bytes; serialization cannot repair a semantic mismatch.

For a generative serving container, define the accepted prompt representation and generation fields from that container’s documentation or implementation. Do not send Bedrock’s messages or inputText fields merely because another inference service uses them. Neither is a universal InvokeEndpoint body field.

Use one explicit encoding for text payloads. Check that the encoded data matches ContentType. A JSON header over CSV bytes is not a format conversion. Likewise, a string containing serialized JSON inside another JSON string changes what the container receives.

Check the synchronous invocation boundary

The InvokeEndpoint API reference documents a maximum request Body length of 6,291,456 bytes, or 6 MiB. Its response Body has the same documented maximum. Measure the final serialized bytes, including framing and encoding overhead, rather than counting source characters or records.

These limits are for the named operation. Do not apply them as universal limits to every SageMaker inference mode. For large payloads or long processing times, SageMaker Asynchronous Inference queues requests with payloads up to 1 GB and processing times up to one hour. Place the payload in Amazon S3 and pass its pointer in InvokeEndpointAsync. SageMaker returns an identifier and output location, then writes the processed result to S3; the invocation response is not the completed prediction.

The API also requires the model container to respond within 60 seconds. AWS recommends a 70-second SDK socket timeout when model processing takes 50–60 seconds. A client timeout setting accommodates the service response boundary; it does not extend the container’s documented processing allowance.

Check size and expected runtime together. A small JSON request can trigger expensive generation, and a large serialized batch can be quick to compute yet fail the body limit. Make both constraints visible in the adapter that approves invocation.

Decode according to the returned type

The service response includes ContentType and Body. ContentType identifies the MIME type returned by the container. Read the returned body using your SDK’s body representation, then decode and parse according to that type and the deployed output schema.

For a container that supports inference streaming, InvokeEndpointWithResponseStream returns the payload incrementally through PayloadPart events. The stream can also contain ModelStreamError or InternalStreamFailure events. Reassemble the parts according to the container’s response framing and handle in-stream errors before validating a complete result; do not assume that each payload part is a complete JSON record.

AWS’s common-format examples show JSON predictions with a predictions array, but the fields within it differ by algorithm. A generative container can define another documented shape. Do not use one prediction parser for every endpoint or assume that an HTTP 200 response contains a plain generated string.

Validate cardinality for multi-record requests: the expected number of results, their association with submitted records, and the documented treatment of per-record failures. Preserve application record identifiers outside any format that cannot carry them. Do not infer correct alignment merely because the returned array has plausible values.

Distinguish a transport failure from a decode failure and from a parsed result that violates the application contract. These require different investigation. A deployment that returns a different MIME type can break decoding even when the model itself produces a valid result.

Preserve invocation identity

CustomAttributes forwards opaque metadata to the container, with a maximum of 1,024 visible US-ASCII characters. A trace identifier is one possible use. The container must set or update that value if you expect it in the response; SageMaker does not automatically echo it as an application acknowledgment.

InferenceId has a different role: when endpoint data capture is enabled, a supplied value is added to captured data. It is not a payload-schema field or a substitute for a business record identifier. Keep tracing, capture identity, and model input separate.

For an endpoint with multiple production variants, TargetVariant overrides routing by variant weights. The response InvokedProductionVariant identifies the variant that handled the call. Use these mechanisms to exercise a serializer and parser against the intended release rather than assuming that a request reached the variant you meant to test.

Key distinctions and trade-offs

Content type versus schema: application/json describes a serialization format. It does not define whether the container expects prompts, tensors, or an instances array. Validate both the representation and the container-specific structure.

Request serialization versus response deserialization: An input adapter can be correct while its output parser is wrong. Release the pair together and verify a complete round trip against representative outputs.

A complete round trip crosses four conversions, two in the application and two in the container.

Payload bytes versus computation time: Body length constrains transferred data; the response deadline constrains synchronous processing. Increasing a client timeout cannot fix an oversized body or authorize extra container processing time.

Tracing versus data capture: CustomAttributes carries container-visible opaque metadata. InferenceId attaches an identifier to captured data when capture is enabled. Neither validates the model’s prediction.

Architecture / decision scenario

A claims pipeline calls two SageMaker AI endpoints. One applies a trained numeric model to claim features; the other hosts a generative container that prepares a review narrative. The team gives each endpoint a separate, versioned serialization contract.

The numeric adapter checks training-time transformations and feature order before generating its documented CSV row. The narrative adapter builds only the JSON fields accepted by its serving container. Both enforce their ContentType and the synchronous byte and runtime boundaries before calling InvokeEndpoint.

A candidate narrative release changes the output object. The team invokes its production variant with TargetVariant, confirms InvokedProductionVariant, and tests the matching response parser. It decodes the returned ContentType, verifies required fields, and rejects incomplete narratives before they reach claim reviewers.

A trace identifier travels through CustomAttributes, and container code explicitly returns it. Where capture is enabled, InferenceId connects the captured invocation to the release evaluation. A malformed result remains a validation failure even if those identifiers match and the HTTP request succeeds.

Common exam traps

• Assuming every SageMaker endpoint accepts one universal JSON schema.
• Declaring JSON while sending CSV bytes.
• Preserving feature values but changing their order or preprocessing.
• Checking source-text length instead of serialized byte length.
• Assuming Accept guarantees that the container implements the requested type.
• Treating a longer client timeout as a longer allowed model runtime.
• Expecting CustomAttributes to be echoed without container code.

What to remember

• Define and validate all four serialization and deserialization boundaries.
• Bind payload shape and MIME types to the deployed model container.
• Preserve training transformations and feature order where applicable.
• Check the named invocation operation’s byte and processing limits.
• Validate returned data before exposing it to downstream consumers.

Knowledge check

References

• AWS, InvokeEndpoint
• AWS, Common data formats for inference
• AWS, Asynchronous inference
• AWS, InvokeEndpointWithResponseStream