NVIDIANVIDIA-Certified Professional: Agentic AI2026-108.1 Define monitoring dashboards and reliability metrics.
Set reliability targets around completed tasks
Completed by the knowledge check
Learning objective
Define SLOs and reliability metrics: availability, latency percentiles, error rate, cost, and task success.
Core idea
Write an SLI specification that separates operational reliability from task success. For an agent service, a reachable endpoint and a completed business task are different outcomes. A fast HTTP 200 response containing an unusable answer can satisfy a transport check while failing the user.
Google’s Site Reliability Engineering book distinguishes a service level indicator (SLI), the quantitative measurement, from a service level objective (SLO), its target. A service level agreement (SLA) adds consequences for missing agreed objectives. Choose targets for the written specification before selecting dashboard panels.
Each specification names the eligible population, good outcome, measurement point, aggregation window, target, and decision owner. This lesson defines those measurements; 8.1.2 connects them to operational dashboards. Serving-performance definitions belong to 4.3.1, and cost-reduction mechanisms to 4.5.2.
How it works
Define the denominator before the percentage
For request-based availability, count eligible user requests that receive a serviceable response divided by all eligible requests. Define “serviceable” explicitly: for example, a complete, structurally valid result before the application deadline. A readiness probe measures a component’s ability to receive traffic; it does not measure that fraction at the user boundary.
Specify how you treat malformed input, permission denials, policy refusals, cancellations, and dependency failures. A correct refusal can be serviceable when refusal is the intended behavior. A dependency timeout remains a user-visible failure even when another team owns the dependency. Exclusions must follow the written eligibility rule, rather than change after an incident to improve the percentage.
Count one logical request once for this SLI. Record retry attempts separately as diagnostic traffic and cost. Otherwise, a request that fails twice and then succeeds produces a different availability depending on which layer counts it. Preserve the request’s terminal outcome and deadline in a team-defined request-outcome ledger.
Separate latency distributions and errors
A percentile describes a distribution: p50 represents the median, while p95 and p99 expose progressively smaller tails. Declare the interval and measurement boundary. For a streaming assistant, measure first useful output and complete-task latency separately; a first token does not establish that retrieval, tool execution, and final validation finished.
Report successful and failed request latency separately, as Google SRE recommends. An overload response that fails quickly can lower an undifferentiated average while the experience deteriorates. Also show the count and fraction of deadline violations across eligible requests, so filtering to successful responses cannot hide timeouts.
Prometheus histograms support aggregation across instances before calculating a percentile with histogram_quantile(). With classic histograms, aggregate compatible buckets and retain the le bucket-boundary label. Do not average replica p99 values. A p99 estimate also depends on bucket resolution; choose boundaries that distinguish the service’s latency target. A threshold-count SLI answers “what fraction finished within eight seconds?” more directly than a percentile estimate.
Define error rate as failed eligible requests divided by eligible requests over the specified window. Keep reason categories such as deadline, tool failure, invalid output, and inference failure so an owner can act. If availability and error rate use identical definitions and populations, they are complements. If they differ, publish the distinction instead of placing apparently contradictory numbers beside each other.
Measure task success and cost independently
Task success asks whether the intended outcome occurred: an answer supports the requested decision, a meeting summary preserves required actions, or an authorized workflow reaches its completion condition. Write the acceptance rubric and assessment coverage into the task-outcome ledger. Evaluation design is covered in 3.1; here you decide how its results enter a production reliability view.
Distinguish verified success, verified failure, pending assessment, and unassessed work. Show assessed successes divided by assessed tasks alongside the assessed share of eligible traffic. Do not label that sample’s success rate as fleet-wide success when the sample omits difficult requests. A request can be operationally serviceable yet fail the task rubric, so retain both outcomes.
Use a cost-allocation ledger for inference, tool calls, retries, and allocated serving infrastructure. Declare the currency, accounting period, included costs, and allocation rule. Report cost per eligible task and cost per verified successful task. The latter exposes configurations that look cheap because they produce many unusable answers. For a vLLM-backed NIM endpoint that exposes them, vllm:prompt_tokens and vllm:generation_tokens count processed prefill and generation tokens. See 3.5.2 for per-step token attribution with the NeMo Agent Toolkit profiler; tokens are not a complete invoice for a self-hosted service.
NVIDIA NIM for LLM and VLM exposes backend inference metrics through /v1/metrics. These describe the serving component. Join them with the application ledgers rather than interpreting inference completion as proof that the business task succeeded.
Tie targets to a decision
Write the measurement window and response policy with each SLO. For a request-based availability target of 99.9%, the allowed bad fraction is 0.1%. A window containing 100,000 eligible requests therefore permits 100 bad requests. This error budget is measured in requests, not automatically in outage minutes.
Use the remaining error budget to decide whether to continue a rollout or prioritize reliability repair. The service owner owns operational availability and latency; the product owner approves the task rubric; the cost owner approves allocation and spend limits. Joint approval prevents one team from improving its metric by transferring failure or expense to another.
Key distinctions and trade-offs
The indicators answer different operational questions:
| Indicator | Deciding question | Measurement artifact or mechanism | Main measurement cost |
|---|---|---|---|
| Availability | Can eligible users obtain a serviceable response? | Request-outcome ledger and good/eligible ratio | Instrumentation at the user boundary |
| Latency percentiles | Are typical and tail responses timely? | Completion timestamps and compatible histogram buckets | Bucket storage and percentile approximation |
| Error rate | Which eligible requests fail, and why? | Terminal-outcome counts with bounded reason categories | Consistent classification across services |
| Cost | Is useful work affordable? | Cost-allocation ledger joined to task outcomes | Shared-cost allocation and delayed billing |
| Task success | Does the result satisfy the intended outcome? | Acceptance rubric and assessed-coverage report | Review effort and delayed assessment |
A single composite score conceals which target failed. Keep separate targets and use a decision rule requiring all mandatory targets to pass. Additional model effort may raise task success while worsening latency and cost; that trade-off needs visible numbers.
Architecture / decision scenario
An insurance research assistant supports claim analysts. Its team sets illustrative 30-day targets: 99.9% request availability, error rate at most 0.1% under the same eligibility rule, completion p95 at most eight seconds, p99 at most 15 seconds, at least 94% assessed task success, and cost at most $0.08 per verified successful task. Assessments use the same rubric and representative sampling plan for both configurations.
Configuration A returns 99.95% serviceable responses from 100,000 eligible requests: 50 errors and a 0.05% error rate. Its p95 is six seconds, p99 is 12 seconds, assessed task success is 90%, and cost is $0.06 per successful task. Configuration B reaches 99.92% availability, with 80 errors and a 0.08% error rate; its p95 is 7.5 seconds, p99 is 14 seconds, success is 95%, and cost is $0.075.
The tempting choice is A because it is faster and cheaper. The product owner rejects that choice because A misses the mandatory task-success target. The service and cost owners approve B: it meets every stated target, although it leaves only 20 bad requests of availability error budget in this measured window. They monitor error-budget consumption before widening deployment rather than treating B’s higher quality as permission to ignore operational degradation.
Common exam traps
- Treating readiness or HTTP success as proof of task success.
- Counting inference retries as independent user requests.
- Averaging replica percentiles instead of aggregating histogram observations.
- Hiding timeouts by reporting latency only for completed requests.
- Reporting assessed success without assessment coverage.
- Equating token usage with total serving cost or an SLO with an SLA.
What to remember
Define eligibility, outcomes, boundaries, windows, targets, and owners before building the dashboard. Availability, latency, errors, cost, and task success describe different consequences. Keep their denominators visible, use the error budget to guide rollout decisions, and require mandatory targets to pass together.
Knowledge check
A research assistant’s inference adapter retries a failed model call up to twice. One user request fails on its first two attempts and succeeds on the third, returning a complete, valid result well before the application deadline. How should the request-based availability SLI count this request?
An agent service runs on six replicas, and each exports a classic Prometheus histogram of completion duration. The SLO report needs one fleet-wide p99 for a five-minute window. How should the team compute it?
le , then apply histogram_quantile()histogram_quantile() on each replica, then average the six resultshistogram_quantile() on each replica, then report the highest resultinstance , then apply histogram_quantile()le gives one fleet distribution to estimate from, whereas an average or a maximum of replica p99 values is not the p99 of the combined traffic, and grouping by instance drops the bucket boundaries and keeps the replicas apart.In one week a claims assistant serves 20,000 eligible requests. Reviewers apply the task rubric to 600 of them, chosen from requests that finished without warnings, and 570 pass. What should the weekly reliability report show for task success?
Two configurations of an underwriting assistant are compared on the same traffic. Configuration A costs $0.04 per eligible task and Configuration B $0.06; under the same rubric and sampling plan, 50% of A’s assessed tasks succeed and 90% of B’s do. The cost target is meant to show whether useful work is affordable. Which measure should it be set against?
A research assistant’s latency report computes p95 only over completed responses. During a dependency incident, many requests hit the 20-second application deadline while others fail fast with overload errors, and the reported p95 improves. Which reporting change exposes the degradation?