Detect coordination faults before remediating them

Learning objective

Specify monitoring for coordination failures, drift, regressions, and automated remediation.

Core idea

Monitor whether the system executes correctly and whether it delivers the right result. A healthy endpoint can return a plausible answer after losing a specialist’s contribution. A technically successful tool call can support a wrong business outcome. For an architect, monitoring must distinguish operational health, coordination correctness, and result quality before an automated action changes the system.

Use the evidence design in 1.2.7 to correlate each failure with its workflow and configuration. This lesson owns the signals, comparisons, alert conditions, and response policy. Instrumentation is useful only when someone can tell which signal warrants investigation, which permits bounded recovery, and which requires a release to stop.

How it works

Observe message delivery and completion

Agent Framework workflow telemetry exposes executor.process, message.send, and edge_group.process spans. The edge_group.delivered boolean indicates successful delivery, while edge_group.delivery_status explains the outcome. Its documented values are delivered, dropped type mismatch, dropped target mismatch, dropped condition false, exception, and buffered.

Interpret the status in the context of the workflow contract. A false routing condition can be an expected branch decision. A type mismatch can reveal a broken executor interface. buffered means the edge is waiting for additional messages in fan-in; it is not itself a failure. Detect a coordination problem when required inputs remain absent beyond the application’s deadline or the workflow reaches a terminal result without its required contributors.

These deadlines and required-contributor sets are application-defined. Add an explicit completion check and emit a bounded custom metric or event when the contract fails. Do not label every non-delivered edge as an error. Conversely, do not count a run as successful merely because a final executor produced output.

Workflow event names workflow.started, workflow.completed, and workflow.error are shared across languages. Their containing span differs: Python uses workflow.run, while .NET uses workflow_invoke. Join executor and delivery evidence with the terminal outcome to distinguish a tool failure, a routing defect, and a fan-in that never completes.

Separate operations from quality

The Foundry Agent Monitoring Dashboard reads the connected Azure Monitor Application Insights resource. Its documented measures include token usage, latency, run success rate, evaluation metrics, and red-teaming results. The portal’s agent metrics view, recurring evaluations, red-team scans, and alerts are marked preview. Do not describe those surfaces as a production service-level guarantee.

Use operational measures to locate pressure and failure, then use evaluation measures to judge outcomes. The dashboard documentation suggests investigation when latency exceeds 10 seconds or run success falls below 95%; these are diagnostic examples, not a universal service objective. Your alert threshold must follow the workload’s agreed latency and success targets, with a time window and enough observations to avoid reacting to noise.

Foundry’s agent evaluators make the quality distinction concrete. Task Completion (builtin.task_completion, preview) measures end-to-end task success. Task Adherence (builtin.task_adherence, preview) measures adherence to instructions and constraints. Tool Call Success (builtin.tool_call_success) measures technical tool execution. A tool-success result cannot stand in for task completion. Keep deterministic business validators alongside evaluator results when the outcome has a strict contract.

Detect drift and regressions with different comparisons

A regression is a deterioration associated with a change, such as a code release, instruction edit, model configuration, or tool schema. Compare candidate and accepted versions against the same controlled dataset, evaluator configuration, and pass criteria. Record the version identifiers so a changed judge or dataset does not masquerade as an agent regression.

Drift is a change in observed production behavior or quality over time. Compare production evaluation results by meaningful cohorts, such as task category or deployed agent version, and inspect the underlying cases. A shift in traffic mix can reduce the aggregate pass rate without a defect in every cohort. Keep a fixed regression dataset as an anchor while expanding coverage for observed production cases. See 1.2.7 for sanitized, versioned datasets and replay evidence.

Foundry distinguishes scheduled evaluations from continuous evaluation of sampled live traffic. In the Python SDK, project_client.evaluation_rules.create_or_update(...) configures an EvaluationRule with ContinuousEvaluationRuleAction, EvaluationRuleEventType.RESPONSE_COMPLETED, and an EvaluationRuleFilter(agent_name=...). The action’s max_hourly_runs bounds evaluation volume. Scheduled runs use project_client.beta.schedules.create_or_update(...) with EvaluationScheduleTask and a RecurrenceTrigger.

Treat these recurring evaluation capabilities as preview. Assign the project managed identity the documented Foundry User role for continuous evaluation and verify the other permissions needed by the chosen workflow. A continuous rule triggered on response completion cannot, by itself, measure work that never completes. Pair it with operational completion and timeout signals.

Connect a condition to a bounded response

Azure Monitor log search alert rules use Custom log search with a Kusto Query Language (KQL) query, a threshold, and evaluation settings. Query the exported coordination signals or application completion metrics, scope the condition to the affected service or version, and choose an observation window. Avoid per-request GUIDs as split-by dimensions: each distinct combination becomes a separate alert instance.

An Azure Monitor action group connects the alert to notifications and automated actions, including an Azure Function, Logic Apps workflow, or webhook. A Secure webhook authenticates to a protected API through Microsoft Entra ID. Action groups execute concurrently without a guaranteed order, so do not model a sequential rollback procedure as several independent actions in one group. Put ordered checks and recovery steps inside the chosen function or workflow.

Define remediation eligibility explicitly. A transient dependency failure might permit a bounded retry; a repeated type mismatch needs a configuration fix; a quality regression might require suspending the candidate release or restoring an approved version. Use an application recovery record keyed to the alert and affected deployment to avoid repeating the same external action. Verify the resulting terminal outcome and evaluator signal before declaring recovery complete. An alert dispatch is evidence that a response was requested, not that the repair succeeded.

Key distinctions and trade-offs

Coordination failure versus expected routing: Delivery statuses need the workflow’s required path and completion deadline to become actionable.

Drift versus regression: Production trends reveal changing behavior; fixed comparative tests reveal deterioration associated with a change.

Run success versus task success: Operational completion and tool execution do not prove that the user’s goal is satisfied.

Detection versus remediation: Azure Monitor identifies a condition; the action group’s target implements and verifies a scoped response. More aggressive recovery reduces delay but increases the risk of repeated side effects or hiding a persistent defect.

Architecture / decision scenario

A procurement workflow requests pricing and compliance reviews before producing an approval recommendation. After a deployment, the endpoint still returns successful responses, but compliance inputs are missing from some recommendations.

The team inspects edge_group.delivery_status and finds dropped type mismatch on the compliance route. The application completion validator rejects results lacking that required contribution and emits its own failure signal. A KQL log search alert scopes the condition to the candidate version instead of restarting every specialist.

A controlled regression run compares the accepted and candidate versions with the same dataset and Task Completion and Task Adherence criteria. Continuous evaluation samples completed production responses to detect quality trends, while timeout monitoring covers incomplete runs. The team labels the Foundry recurring evaluation and dashboard features as preview in its operating plan.

An action group invokes one recovery workflow that checks the affected version, suspends its rollout, restores the approved configuration if authorized, and records the recovery outcome. It does not blindly retry a type mismatch. Operators verify restored delivery, completion checks, and evaluation results before resuming the rollout, preserving evidence for a code fix.

Common exam traps

• Treating buffered or dropped condition false as an unconditional failure.
• Using tool execution success as the task-quality measure.
• Detecting never-completed work only through RESPONSE_COMPLETED evaluation rules.
• Comparing releases after changing the dataset or evaluator without accounting for that change.
• Presenting Foundry monitoring and recurring evaluation preview features as guaranteed production controls.
• Assuming concurrent action-group actions form an ordered repair sequence.

What to remember

Observe routing, required contributions, completion, and quality separately. Use fixed evaluation comparisons for regressions and sampled production evidence for drift. Turn a scoped alert into one bounded, verifiable recovery procedure, and escalate defects that another execution cannot repair.

Knowledge check

References

• Microsoft, Agent Framework workflow observability
• Microsoft, Monitor agents with the Agent Monitoring Dashboard
• Microsoft, Agent evaluators
• Microsoft, Create a log search alert rule
• Microsoft, Action groups
• Microsoft, Task Completion evaluator sample
• Microsoft, Tool Call Success evaluator sample