Recover actions without duplicating their effects

Learning objective

Design fallbacks, idempotent actions, and compensating steps for partially completed side effects.

Core idea

An agent workflow can fail after some external actions have succeeded. Returning an error does not reverse those actions, and rerunning the whole workflow can duplicate them. Recovery starts by determining which effects are confirmed, which never happened, and which remain uncertain.

Use fallback when an approved alternative can still satisfy the request. Use idempotency when an action might be repeated. Use compensation when confirmed completed work needs a business-specific reversal. These mechanisms cooperate, but none replaces an accurate operation record. See 2.4.2 for bounded retries and 2.4.3 for dependency admission; this lesson decides what happens to external state after those controls stop execution.

How it works

Record the action before deciding recovery

Microsoft’s Compensating Transaction pattern describes a workflow recording completed steps and information needed to undo them. Compensation uses that record after failure. Design the recovery record as a durable application artifact rather than relying on the agent’s conversational recollection.

For example, define application fields operation_id, step_id, request_fingerprint, status, provider_reference, and compensation_status. These are proposed domain fields, not NeMo Agent Toolkit configuration keys. The logical operation identifier remains stable across attempts. The provider reference connects the local record to the external action that actually occurred.

A useful application state vocabulary distinguishes planned, in_progress, confirmed, failed, and unknown. The exact names are your schema choice. A timeout after submission belongs in unknown until a provider status query or reconciliation process settles it. Do not record failed merely because the caller stopped waiting.

For every side-effecting step, specify the accepted confirmation, permitted replay mechanism, status-query path, and compensating operation before production use. If the provider offers no way to resolve an uncertain write safely, the recovery policy must include an operator-owned unresolved outcome. A model-generated assertion cannot substitute for the provider’s state.

Preserve intent across retries

An idempotent action has the same intended effect when repeated as when applied once. Microsoft’s Retry pattern highlights why this matters: a service can process a request successfully and lose the response, causing a caller to repeat work that already happened.

For a tool you own, design the logical operation identifier to select an existing result rather than create another effect. For an external API, use its documented idempotency mechanism where one exists, keeping its key and request meaning stable across attempts. If a repeated key arrives with different parameters, reject the mismatch instead of treating a changed instruction as a retry. The stored request_fingerprint makes that check explicit in the proposed application design.

Keep the deduplication decision at the authority that owns the effect. Recording an operation identifier in the agent’s local history cannot by itself prevent two remote bookings. Likewise, writing a local success marker after a remote call leaves a crash window between the external effect and that marker. Use a provider reference and reconciliation to handle that window rather than claiming that a local record makes the distributed action atomic.

Choose a fallback that preserves the request

Fallback is an alternative execution path or an explicit reduced service. Examples include an approved alternate provider or a draft itinerary when confirmed booking is unavailable. The choice must preserve authorization, processing location, accepted output quality, and the user’s permitted side effects.

Do not use fallback to escape a business rejection or to hide an unknown write. Before switching providers after a booking timeout, settle whether the original reservation exists. Otherwise, a successful fallback can leave the user with two confirmed reservations.

Give the fallback the remaining operation budget and identify its result as degraded when it does less than requested. Microsoft’s Circuit Breaker pattern permits a meaningful default result in the open state; that does not make every default equivalent to the requested operation. The application decides what result is acceptable and what requires explicit confirmation from the user.

Compensate confirmed work and resume recovery

Compensation is a business action that counteracts a completed step, such as releasing a reservation. It is not necessarily a database rollback or a restoration of the entire prior world. Microsoft’s pattern warns that restoring a snapshot can overwrite concurrent changes. Cancel the reservation owned by this operation using its provider reference; do not reset the provider’s entire inventory to an earlier value.

The pattern also states that compensation need not run in the exact reverse order, and some recovery actions can run in parallel. Choose order from dependencies and business risk. A scarce resource might need release before a lower-impact notification is corrected.

Recovery can fail, so persist progress and resume from the failed compensating step. Design each compensating command to be idempotent so repeating recovery does not duplicate its effects. Distinguish a compensated workflow from a completed original request. If an effect cannot be reversed, choose a corrective action or manual intervention and retain enough context for the owner to finish it. A sent email, for example, may require a correction rather than an attempt to erase the recipient’s copy.

Make the NVIDIA workflow report the right outcome

NeMo Agent Toolkit’s workflow._type: sequential_executor executes its tool_list in order, passing each output to the next function. Its return_error_on_exception defaults to False, which re-raises exceptions. With True, it exits early and returns an error message as workflow output. That choice controls error propagation; implement the recovery record and business reversal in the application functions.

SequentialExecutorExit supports intentional early termination with a custom output message. Use that distinction when an application recovery step deliberately stops normal work to report an unresolved or compensated outcome. Do not equate an error message returned by the executor with successful completion of the original task.

Key distinctions and trade-offs

Choose among these mechanisms from the known external state:

Idempotency prevents duplicate effects; compensation counteracts effects already applied. Fallback chooses another path. Combining their names does not make a workflow transactional across independent services.

Architecture / decision scenario

An employee travel assistant has 120 seconds to confirm a rail reservation and hotel booking, then deliver an itinerary. A rail reservation is confirmed after 15 seconds. The hotel request times out at 45 seconds without a confirmation response. The travel team requires at most one hotel booking and permits a draft itinerary as reduced service.

A proposed fallback immediately books a second hotel through another provider. The application architect rejects it because the first hotel’s outcome is unknown. The recovery function retains the hotel operation identifier and queries the provider’s status. If the booking is confirmed, the workflow continues without replay. If it definitively failed and enough time remains, the approved fallback can book an alternative using its own recorded step identity.

If no hotel can be confirmed before the deadline, the travel policy requires release of the confirmed rail reservation. The compensating function cancels that reservation by its provider reference and records completion. If cancellation also times out, recovery remains unresolved and is assigned to travel operations; the assistant supplies a draft and describes the reservation uncertainty. After a restart, the worker checks the cancellation status using the provider reference before resuming; if cancellation still needs a retry, it reuses the same recorded compensating-step identity. It does not book another train. The travel owner defines acceptable alternatives and cancellation rules; the service owner maintains safe replay and reconciliation.

Common exam traps

• Treating a local timeout as proof that a remote write failed.
• Generating another operation key for every retry.
• Switching providers before resolving the first provider’s uncertain result.
• Assuming compensation restores the exact prior state or always runs in reverse order.
• Forgetting that compensating actions can fail and need resumable progress.
• Treating an executor’s returned error message as successful business completion.

What to remember

Recover from recorded external state, not from the last exception alone. Preserve operation identity, resolve ambiguous writes before fallback, and compensate only the confirmed effects that business policy requires. Recovery has its own progress, failures, and owner.

Knowledge check

References

• NVIDIA, Configure With the Sequential Executor
• Microsoft, Retry pattern
• Microsoft, Circuit Breaker pattern
• Microsoft, Compensating Transaction pattern