Layer unit, integration, and regression agent tests

Learning objective

Write unit, integration, and regression tests across tools, agents, and workflows.

Core idea

A multi-agent solution needs tests at several levels because a correct final answer can hide an incorrect tool operation, and a correct tool implementation can still be called with the wrong arguments. Test the component’s contract, the connections between components, and the behavior of the compatible release set separately.

Unit and integration describe the scope of execution. Regression describes the purpose: proving that a change preserves required behavior. A regression case can therefore be a unit test for a tool, an integration test for a specialist handoff, or a dataset-driven evaluation of the complete workflow. These are complementary axes, not three mutually exclusive test folders.

Microsoft Foundry’s hosted-agent testing guide distinguishes unit tests, local integration, deployed integration, and structured evaluation. Use that distinction to allocate fast feedback and realistic evidence. A few successful interactive prompts do not establish that a candidate preserves behavior across previously failing cases.

How it works

Test tools and deterministic agent logic first

Microsoft’s hosted-agent guide uses pytest for Python and xUnit for .NET. It recommends testing core logic without running the full server or calling external models. Tool handlers, prompt formatting, and agent logic are explicit unit-test targets; pytest test_agent.py and dotnet test are the corresponding example runner commands.

For a purchase tool, assert required argument validation, tenant-specific lookup, amount calculation, and the returned result structure. Substitute the remote purchasing service so the test cannot charge an account. Test rejection paths as carefully as success: a missing order identifier should fail before any side effect. These assertions describe your application’s contract, rather than a vendor guarantee about a tool.

At the agent level, supply controlled model or tool responses to test how the application handles a requested operation. Check that an invalid argument becomes the expected failure, that a refusal is preserved, and that the next workflow participant receives the required fields. The same isolation principle applies to workflow routing and state transformations: injected outcomes let you exercise paths that a live model might rarely select.

Avoid making a live language-model call a prerequisite for every unit test. A model outage should not obscure a deterministic calculation defect. Conversely, a passing substitute-based test proves only the contract exercised by that substitute; it does not prove that the deployed model or remote tool implements the same contract.

Exercise local and deployed connections

For hosted agents, azd ai agent run starts local execution, and azd ai agent invoke --local sends requests to it. The testing guide also documents the local /readiness probe, expecting HTTP 200, and the /responses endpoint. These are hosted-agent testing mechanisms, not universal endpoints for every application using Agent Framework.

Local integration includes real model calls and protocol handling. Test whether a tool definition reaches the model, returned arguments reach the handler, and tool results reach the agent’s next turn. At workflow scope, execute the handoff and final result assembly with the participants connected. A unit-tested transformation can still fail when another participant sends a differently shaped message.

Deployed integration uses azd ai agent invoke without --local. It adds Foundry infrastructure and the network path to the evidence. Exercise deployed credentials, tool connectivity, and response handling through the identity and resources assigned to that environment. See 4.3.2 for environment isolation; local success cannot substitute for this test.

Use azd ai agent invoke --new-session when a case needs a fresh conversation. Also include intentional multi-turn cases: session isolation and continuation test different contracts. Arrange fixtures so a previous test’s history cannot accidentally supply information that the candidate should retrieve for itself. Record whether each case requires clean state or deliberate prior context.

Preserve incidents as regression cases

A regression suite fixes the input, expected outcome, and evidence needed to judge the change. Preserve a tool timeout that previously caused duplicate submission, a handoff that lost an identifier, and an agent answer that claimed an operation succeeded after the tool failed. Keep the cases reusable without replaying production side effects.

Exact assertions work for deterministic fields and prohibited operations. Semantic output needs a rubric or evaluator rather than equality with one saved paragraph. See 3.1.3 for response regression criteria and 3.1.4 for tool-trajectory validation. Here the test design must retain both the final result and the tool interaction needed to assess those criteria.

The hosted-agent guide provides azd ai agent eval generate to create a small smoke dataset, a default adaptive evaluator, and eval.yaml. Run a recipe with azd ai agent eval run --config eval.yaml and inspect it with azd ai agent eval show --eval-run-id. Expand the generated smoke cases into your regression corpus; their small size is not evidence of production coverage.

Microsoft states that azd ai agent eval suites is not yet generally available. Version eval.yaml alongside the agent code instead of designing CI around that subcommand. Release acceptance from completed evaluation evidence belongs to 4.3.7.

Make failures visible to CI

Azure Pipelines can run pytest with --junitxml and publish its output using PublishTestResults@2; see 0.14 for the Python pipeline. The task’s testResultsFormat identifies the result format, and testResultsFiles identifies the files to collect. Publishing a report is a separate operation from deciding that a failed test must block the candidate.

Set failTaskOnFailedTests: true when published failures must fail the task. failTaskOnMissingResultsFile and failTaskOnFailureToPublishResults cover missing evidence and publication failures; their documented defaults are false, as is failTaskOnFailedTests. Preserve runner failures too. A pipeline that loses the test runner’s failure and merely uploads XML can appear successful despite failing cases.

Key distinctions and trade-offs

Choose scope according to the question you need answered:

Fast tests localize defects cheaply. Real integrations cost time and can encounter service failures, but reveal mismatches that substitutes hide. Keep both instead of increasing model-backed coverage until every test is slow and difficult to diagnose.

Architecture / decision scenario

A procurement workflow delegates product lookup to one agent and purchase preparation to another. A tool update changes the expected currency field while a prompt update changes how the agents hand off the selected product.

Tool unit tests assert argument validation and the resulting amount structure without submitting an order. Agent unit tests inject a failed lookup and verify that the application does not continue with a fabricated product. Workflow unit tests verify that a handoff retains the product and tenant identifiers.

Local integration exercises the connected agents against the real model. Deployed integration invokes the candidate in a fresh session and verifies tool connectivity under its assigned identity. A regression corpus includes the earlier missing-currency incident and a multi-turn case that corrects the selected product. The evaluation recipe judges the answer and the tool trajectory. CI retains the runner failure status and fails publication when required evidence is missing; a polished final paragraph cannot override a wrong purchase argument.

Common exam traps

• Treating regression as a separate execution level that excludes unit tests.
• Calling a live-model integration test a deterministic unit test.
• Checking only the final answer when tools can create side effects.
• Using local invoke results as proof of deployed credentials and connectivity.
• Treating a generated smoke dataset as a complete regression corpus.
• Assuming PublishTestResults@2 fails on test failures by default.

What to remember

Test tool contracts, agent decisions, and connected workflows separately. Preserve failure cases across those levels as regression coverage, isolate conversation state deliberately, and retain tool evidence as well as final answers. A release needs executable failures and complete reports, not only a successful invocation.

Knowledge check

References

• Microsoft, Test a hosted agent
• Microsoft, Customize Python pipelines
• Microsoft, PublishTestResults@2 - Publish Test Results v2 task