Configure Model Armor runtime controls

Learning objective

Configure Model Armor to inspect prompts and model responses, tune confidence thresholds, and choose between observation and active blocking. This lesson focuses on runtime threats such as prompt injection, jailbreak attempts, harmful content, and malicious URLs. Sensitive Data Protection configuration belongs to lesson 5.2.6.

Core idea

Treat Model Armor configuration as three independent policy choices:

enabled filters -> confidence threshold -> enforcement mode
what to inspect    what is a match        what happens to a match

A filter can run without blocking traffic. A strict enforcement mode cannot detect a threat whose filter is disabled, and a filter does not report a match unless its result meets the configured threshold. Keeping these choices separate makes a runtime policy easier to tune and audit.

Place Model Armor on both sides of the model:

user prompt -> inspect -> model -> inspect -> user or downstream consumer

Call sanitizeUserPrompt before sending input to the model and sanitizeModelResponse before releasing output. The Model Armor overview recommends separate input and output templates because the two boundaries have different threats. An input template might emphasize prompt injection and jailbreak attempts. An output template might emphasize harmful content and malicious URLs.

How it works

A template defines the filters, supported thresholds, and enforcement metadata. Model Armor provides Responsible AI filters for categories such as hate speech, harassment, sexually explicit content, and dangerous content. It also provides prompt-injection and jailbreak detection, malicious-URL detection, and Sensitive Data Protection integration. The child sexual abuse material (CSAM) filter is enabled by default and cannot be disabled. See Model Armor filters for the available categories and limits.

Filter enablement and threshold configuration are not the same setting. The prompt-injection and jailbreak detector, for example, has both an ENABLED/DISABLED setting and a confidence level. Malicious-URL detection has an enablement setting but no confidence threshold in the template schema.

The following partial template makes these choices explicit:

{
  "filterConfig": {
    "piAndJailbreakFilterSettings": {
      "filterEnforcement": "ENABLED",
      "confidenceLevel": "HIGH"
    },
    "maliciousUriFilterSettings": {
      "filterEnforcement": "ENABLED"
    }
  },
  "templateMetadata": {
    "enforcementType": "INSPECT_ONLY"
  }
}

Add the required category-specific Responsible AI filters to the same template. The template configuration guide documents the complete configuration and creation methods.

For filters that support granular confidence thresholds, a result matches when the detected confidence is equal to or higher than the configured value:

Lowering the threshold makes the detector more sensitive. It does not weaken the policy. Configure thresholds explicitly instead of relying on defaults, and tune each category against representative safe and adversarial traffic. The confidence-level guidance explains why one threshold is not appropriate for every category.

The template-wide enforcement type determines what happens after a match:

• INSPECT_ONLY runs the configured filters but lets the interaction continue. Use it to measure detections and false positives before changing live traffic.
• INSPECT_AND_BLOCK produces an active block verdict. A violating prompt must stop before the model, and a violating response must stop before the user or another consumer.

The template default is INSPECT_AND_BLOCK, but set the value explicitly so reviewers can see the intended behavior. In custom integrations, the calling service is the policy enforcement point: it must branch on the verdict. An API call that records a match and then continues processing has not enforced the policy. Integrated services can perform this enforcement inline; check the behavior of the chosen Model Armor integration.

Read two result dimensions before continuing. filterMatchState reports whether at least one configured filter matched. invocationResult reports whether all filters ran successfully: SUCCESS, PARTIAL, or FAILURE. A NO_MATCH_FOUND result is not proof of safety when a required filter was skipped or failed. The SanitizationResult reference defines both fields and the per-filter execution details.

A match stops traffic only under a block verdict, and no match counts only when every filter ran.

Key distinctions and trade-offs

Start a new policy in INSPECT_ONLY, enable sanitization logging, and protect the logs as sensitive data. Model Armor sanitization logs can contain full prompts, responses, and filter results. Measure false-positive and false-negative costs, tune individual thresholds, and then change the template to INSPECT_AND_BLOCK. Continue monitoring after deployment because traffic and attack patterns change. See Configure logging for the logged data.

Use floor settings when an organization needs minimum filter and threshold requirements across the resource hierarchy. Floor settings reduce the chance that an application owner creates an overly permissive template. They are a governance baseline, not a substitute for placing and enforcing the runtime checks. See Configure floor settings.

Model Armor is also not an authorization control. A successful content scan does not grant permission to call a tool, reveal data, or perform a side effect. Keep identity, authorization, tool validation, and human approval controls in place.

Architecture / decision scenario

A customer-support agent can search internal documentation and draft a reply. An approved workflow later sends that reply to a customer.

The input template enables prompt-injection and jailbreak detection at MEDIUM_AND_ABOVE and harmful-content categories at HIGH. The output template enables malicious-URL detection and uses category-specific harmful-content thresholds. The team first deploys both templates in INSPECT_ONLY, protects the resulting logs, and tests normal support questions, quoted attack text, known injection attempts, and unsafe generated links.

After review, the team changes both templates to INSPECT_AND_BLOCK. The application stops a matched prompt before model invocation and replaces a blocked output with a generic safe message. It also denies the send operation when a required detector returns PARTIAL or FAILURE. Tool authorization still decides whether the agent may send a reply; Model Armor decides only whether the inspected content violates its configured policy.

Common exam traps

• Assuming HIGH is the most aggressive threshold. LOW_AND_ABOVE flags more content and usually creates more false positives.
• Confusing filter enablement, confidence thresholds, and enforcement mode.
• Using INSPECT_ONLY as protection rather than as observation and tuning.
• Inspecting prompts but releasing model responses without inspection.
• Treating NO_MATCH_FOUND as safe without checking invocationResult and required per-filter execution states.
• Assuming a custom application blocks automatically after receiving a verdict.
• Treating Model Armor as a replacement for authorization or tool-side policy.

What to remember

• Enable the filters that match the threat at each runtime boundary.
• Set explicit, category-specific thresholds and calibrate them with real data.
• Use INSPECT_ONLY to observe, then INSPECT_AND_BLOCK for active enforcement.
• Inspect both prompts and responses, and make the integration enforce verdicts.
• Handle detector failures separately from no-match results.
• Keep authorization and side-effect controls outside the content filter.

Knowledge check

References

• Google Cloud, Model Armor overview
• Google Cloud, Create and manage templates
• Google Cloud, Sanitize prompts and responses
• Google Cloud, SanitizationResult REST reference
• Google Cloud, Configure Model Armor floor settings