Skip to lesson

NVIDIANVIDIA-Certified Professional: Agentic AI2026-100 Prerequisites

Bound async calls and classify failures

Completed by the knowledge check

Learning objective

Call OpenAI-compatible LLM endpoints from Python with async I/O and structured error handling.

Core idea

An asynchronous call lets a Python task yield while it waits for network I/O. It does not make model inference faster or create GPU capacity. Your integration must still select the endpoint, bound its wait, and translate failures into outcomes the caller can handle.

OpenAI compatibility describes an API interface. NVIDIA NIM for LLM and VLM documents POST /v1/chat/completions and GET /v1/models; use the deployed model’s identifier rather than assuming a model name from another provider works. See 7.2.1 for NIM startup, health validation, and API calls.

How it works

The OpenAI Python SDK exposes AsyncOpenAI. Set base_url to the approved API root ending in /v1, supply the credential required by that deployment, and await client.chat.completions.create(...). An async with block closes the client when work finishes.

This non-streaming example accepts deployment-specific values and returns a small outcome object:

import asyncio
import openai

async def ask(base_url: str, api_key: str, model: str, prompt: str):
    async with openai.AsyncOpenAI(
        base_url=base_url, api_key=api_key,
        timeout=8.0, max_retries=0,
    ) as client:
        try:
            async with asyncio.timeout(8.0):
                reply = await client.chat.completions.create(
                    model=model,
                    messages=[{"role": "user", "content": prompt}],
                )
            return {"kind": "response", "response": reply}
        except TimeoutError:
            return {"kind": "deadline_exceeded"}
        except openai.APITimeoutError:
            return {"kind": "timeout"}
        except openai.APIConnectionError:
            return {"kind": "connection_failure"}
        except openai.APIStatusError as error:
            return {"kind": "http_error", "status": error.status_code}

Call ask(...) with await from an asynchronous entry point on Python 3.11 or later. The eight-second application allowance is an example setting, not a NIM limit. max_retries=0 makes this one SDK attempt; it does not disable retries elsewhere in your system.

The exception order matters: APITimeoutError is a connection-error subtype. APIStatusError means an HTTP error response arrived and exposes status_code; RateLimitError represents HTTP 429. A timeout means you stopped waiting, not that the server never processed the request. Keep the response object for application validation rather than equating receipt with task success.

Key distinctions and trade-offs

The SDK timeout controls network waits, including connection, read, and write waits; it is not a total wall-clock deadline. The example uses asyncio.timeout(8.0) around the call to enforce its application allowance by cancelling overdue work and raising TimeoutError outside the context manager. Preserve cancellation after cleanup rather than swallowing asyncio.CancelledError.

The application allowance wraps the one SDK attempt: the attempt ends as APITimeoutError, APIConnectionError or APIStatusError, or as a response kept for application validation, and overdue work ends as TimeoutError outside it.

The application allowance wraps one SDK attempt and raises TimeoutError outside itInside the application allowance, one SDK attempt ends either as APITimeoutError, APIConnectionError or APIStatusError, or as a response object, and only the response object goes on to application validation. When the allowance cancels overdue work, the outcome is TimeoutError, raised outside the allowance.application allowancecancelling overdue workone SDK attemptAPITimeoutError,APIConnectionErroror APIStatusErrorresponse objectTimeoutErrorapplicationvalidation

For concurrent calls, Python 3.11+ provides asyncio.TaskGroup and its create_task(...) method. A shared asyncio.Semaphore acquired with async with around each call caps the requests in flight to one inference service. Set that cap from the service’s tested capacity rather than creating unbounded tasks.

The SDK retries eligible failures twice by default. Explicit retry settings prevent an unnoticed SDK policy from multiplying an outer retry loop. See 2.4.1 for failure classification, 2.4.2 for bounded retry design, and 2.4.4 for side effects and idempotency. A broad except Exception that turns everything into empty text discards the distinctions those mechanisms require.

Architecture / decision scenario

A logistics employee assistant has a ten-second response budget, with two seconds reserved for authorization and rendering. The integration owner uses the example eight-second asyncio.timeout(...) allowance around inference and disables SDK retries while the application owns the overall deadline.

A colleague proposes three sequential attempts to hide intermittent failures. If each attempt exhausts its eight-second application allowance, three attempts consume at least 24 seconds before other work, so that proposal violates the budget. The owner returns a typed unavailable outcome after a connection failure, surfaces HTTP 400 for input correction, and sends 429 to the bounded-retry policy only when remaining time permits. The owner caps concurrent calls with a shared asyncio.Semaphore; adding coroutines cannot resolve this deadline conflict.

Common exam traps

  • Treating async as GPU parallelism or unlimited admission capacity.
  • Assuming a timeout proves the server did no work.
  • Retrying every HTTP error unchanged or hiding it as an empty answer.
  • Forgetting retries already configured in the SDK.

What to remember

Use AsyncOpenAI, an explicit deployment URL, and await. Keep timeout, connection failure, HTTP failure, and response validation separate. Retry policy must fit the caller’s total deadline.

Knowledge check

References