Turn recurring failures into bounded recovery

Learning objective

Trigger alerts, escalation, and automated remediation for recurring failure signatures

Core idea

A recurring failure signature is a stable description of a repeated problem: the same contract mismatch, failing dependency, or coordination outcome in a defined component and version. Detecting the signature, notifying an owner, and performing a recovery action are separate operations. An alert proves a configured condition was met; it does not prove that repeating the original work is safe.

For architects and senior engineers, Azure Monitor supplies the detection and action dispatch mechanisms. Your recovery handler enforces authorization, deduplication, and the permitted scope of change. See 1.2.8 for the overall monitoring and recovery architecture and 2.3.8 for retry budgets. This lesson focuses on making repeated failure patterns actionable without turning an alert storm into a recovery storm.

How it works

Define a stable failure signature

Foundry Monitor settings provide Alerts (preview) for agent latency, token usage, evaluation scores, and red team findings. This built-in option detects agent performance anomalies, evaluation failures, and security risks. For a recurring custom failure signature, use an Azure Monitor log search alert rule to control query grouping, Number of violations, Split by dimensions, and action-group dispatch.

Build a Kusto Query Language (KQL) query over the telemetry collected in Application Insights or its Log Analytics workspace. Group by stable fields such as agent role, deployed version, dependency name, and failure class. If these are custom application properties, document their schema and query mapping explicitly. Do not present your signature field names as built-in Azure Monitor attributes.

For coordination failures, the Agent Framework routing field edge_group.delivery_status provides documented outcomes. dropped type mismatch indicates a different repair from exception; buffered can be ordinary fan-in waiting rather than a defect. See 1.2.8 for routing-status interpretation and 3.3.1 for aggregate coordination counts when choosing records for the alert query.

Keep individual request and trace identifiers in drill-down evidence. A unique trace ID is useful for investigation but poor for recurrence grouping: every occurrence would become its own series. Normalize failure classes rather than grouping on arbitrary exception text that contains changing identifiers or payload fragments.

Configure the log search alert

Azure Monitor log search alert rules combine scope, query, measurement, and triggering conditions. Aggregation granularity defines the window used to aggregate records. Frequency of evaluation defines how often the query runs. Threshold, Operator, and Threshold value express the condition that should fire.

For recurrence, use Number of violations and Evaluation period under the advanced options. The documented example combines a five-minute aggregation granularity with three violations in the last hour. Those are example settings, not a universal incident policy. Choose values according to how quickly the failure threatens callers and how much transient noise you can tolerate.

The documentation requires a datetime column in query results for the number-of-violations option. Preserve the temporal structure needed by the alert instead of reducing the query to a timeless grand total. Also distinguish a violation from a failed request: a single violating window can contain many failing requests.

Use Split by dimensions when different agents or failure classes need independent alert instances. Azure Monitor supports up to six dimensions per log search alert rule, with string or numeric columns. Each unique combination is evaluated separately. Avoid timestamps, GUIDs, and other volatile values because they generate separate instances and undermine suppression across evaluations.

Dispatch notification and escalation

Attach an Azure Monitor action group to the alert. Action groups define notifications and automated actions; supported actions include an HTTP-triggered Azure Function, a webhook, and a Logic Apps workflow. An alert rule can reference up to five action groups, and they execute concurrently in no specified order.

Concurrency changes the design. Do not make one action group’s notification a prerequisite for another group’s recovery step. If escalation depends on the outcome of a recovery attempt, implement that sequence in one handler or Logic Apps workflow, then send the resulting notification from that sequence.

Log search alert rules from API version 2020-05-01 support only the common alert schema. For other alert types, enable that schema on the chosen action integration. Its schemaId identifies azureMonitorCommonAlertSchema. The data.essentials object includes alertId, severity, monitorCondition, alertTargetIDs, and firedDateTime. monitorCondition is Fired when the condition triggers and Resolved when it clears. Parse these fields rather than treating every delivered payload as a fresh outage.

Route the signature to an owner and severity defined in your incident policy. A Logic Apps workflow can customize the notification and escalation integration. The presence of an action group alone does not establish an acknowledgment deadline or a human escalation ladder; those remain explicit application and operations responsibilities.

Guard the automated action

Before changing anything, the recovery handler checks the alert’s affected resource, condition, and allowed action. Use alertId with your application incident state to deduplicate handling, and record the action result against the same incident. This is a proposed handler design, not an Azure Monitor exactly-once guarantee.

Recheck that the failure still exists and that the target version matches the version implicated by the signature. An alert can arrive after the application has recovered or after a deployment changed the target. Separate a Resolved notification from authorization to initiate another recovery attempt.

Permit only bounded, preapproved actions. A contract mismatch can trigger escalation and a controlled release rollback process; blindly retrying the incompatible message does not repair its type. A transient dependency failure can trigger a constrained recovery path that shares the request’s existing budgets. Preserve the validation and side-effect rules from 2.3.8–2.3.10.

Record remediation start, result, and verification as correlated application telemetry. If recovery fails or the signature recurs beyond the incident policy’s limit, stop automated attempts and escalate. Suppressing notifications is different from resolving the underlying condition; the service needs post-action evidence of restored behavior.

The action group dispatches a Logic Apps workflow and the recovery handler, and only the handler’s checks admit a bounded action.

Key distinctions and trade-offs

Threshold crossing versus recurrence: A single evaluation can detect a spike. Multiple violations in an evaluation period establish repeated breaches of the configured condition, not necessarily repeated identical requests.

Alert instance versus recovery attempt: alertId identifies the alert instance. Your handler controls how many recovery attempts that incident permits and what has already happened.

Notification versus orchestration: Concurrent action groups dispatch actions. A handler or Logic Apps workflow owns a sequence that requires ordering, checks, or acknowledgment.

Faster alerts reduce detection delay but increase sensitivity to transient noise. More dimensions improve ownership but multiply alert instances. Choose stable signatures and a bounded recovery contract before tuning notification volume.

Architecture / decision scenario

The incident-summary service repeatedly drops classification messages after a release. A KQL query groups dropped type mismatch by the application role and version fields. The log search rule uses a five-minute aggregation window and the team’s recurrence policy of three violations within an hour, preserving the datetime output required by that option.

An action group invokes a Logic Apps escalation workflow and an HTTP-triggered Azure Function. Because actions run concurrently, the function performs its own authorization and deduplication checks using the common schema’s alertId and monitorCondition. It does not assume the human notification ran first.

The handler detects an incompatible contract and declines message replay. It records the verified signature, stops bounded recovery, and routes the incident to the release owner for the approved correction. For a separate transient dependency signature, the same design permits a constrained recovery operation, verifies the affected path afterward, and escalates if recurrence continues. Alerting, escalation, and remediation each produce evidence rather than being treated as one successful operation.

Common exam traps

• Splitting recurrence alerts by unique trace IDs or changing timestamps.
• Confusing three violating windows with exactly three failed requests.
• Assuming action groups execute in a dependable order.
• Running recovery for both Fired and Resolved payloads unchanged.
• Treating notification suppression as evidence that the defect is fixed.
• Replaying incompatible or consequential work merely because an alert fired.

What to remember

Use a stable query signature, explicit window and recurrence settings, and action groups for dispatch. Parse the common alert schema and enforce bounded recovery in the handler. Verify restoration, record the outcome, and escalate when the permitted automated path cannot repair the failure.

Knowledge check

References

• Microsoft, Monitor agents with the Agent Monitoring Dashboard
• Microsoft, Create Azure Monitor log search alert rules
• Microsoft, Create and manage action groups in Azure Monitor
• Microsoft, Common alert schema
• Microsoft, Microsoft Agent Framework workflows: Observability