# Read-Only Is a Trust Level, Not a Feeling: How to Govern AI Ops Agents Before They Remediate Production

Enterprise teams are understandably interested in AI incident response, especially when an agent can quickly analyze live data and suggest a root cause, as demonstrated in Microsoft’s [Azure Ops Workbench War Room walkthrough](https://techcommunity.microsoft.com/blog/microsoftmissioncriticalblog/building-an-ai-azure-ops-workbench-%E2%80%94-part-2-the-war-room-%E2%80%94-watch-ai-root-cause-a/4540765). However, the ability to investigate production doesn't mean an agent should automatically make changes. A common approach is to separate read-only agents from those that can make changes, using policy boundaries enforced at every MCP tool call.

As platforms expand, this separation becomes crucial. The [Construct platform checklist](https://construct.computer/blog/how-to-choose-an-ai-agent-platform-for-your-team/) stresses that governance relies on tangible mechanisms like human-in-the-loop controls and activity inspection, not just marketing claims. The [Google Workspace MCP guidance](https://developers.google.com/workspace/guides/configure-mcp-servers) also highlights that permissions and behavior depend on server and OAuth scope configuration, underscoring the need for explicit and enforced tool trust levels.

## 1) Ops-agent lifecycle: query -> correlate -> hypothesize -> validate -> recommend -> remediate

A well-managed ops agent follows a clear lifecycle. It begins with **query**, gathering evidence from sources like Azure Monitor, Key Vault diagnostics, Kubernetes events, CI/CD logs, and Entra ID activity. It then **correlates** these signals across various parameters.

Next, in **hypothesize**, the agent suggests possible causes and ranks them by likelihood. During **validate**, it performs tests to challenge its top hypothesis before increasing confidence. Only then should it **recommend** a change, detailing the expected impact and rollback plan.

The final stage, **remediate**, should be treated as a separate trust boundary, not an automatic follow-up to diagnostics. Many incidents should conclude at “recommend” unless policy and approval allow for changes.

## 2) Trust taxonomy with practical tiers

Trust should be based on operation class rather than subjective assessments of agent behavior. The table below outlines a practical taxonomy applicable to Azure MCP, Microsoft Graph MCP, GitHub, CI/CD, and Kubernetes tools.

| Trust tier | Typical MCP/tool operations | Operational risk | Default control posture |
| --- | --- | --- | --- |
| **T0 Inspect/List/Read** | `list`, `get`, `read`, log query, metric query, config fetch | Low, mostly observational | Allow in prod for approved agent identities; full logging required |
| **T1 Diagnose/Profile/Test** | profiling, dry-run tests, diff checks, canary analysis, policy simulation | Low-to-medium (can consume resources, may expose sensitive context) | Allow with guardrails (rate, scope, data minimization) |
| **T2 Create Ticket/Draft Change** | create incident ticket, open PR draft, generate remediation plan, draft runbook steps | Medium (workflow side effects, but no direct infra mutation) | Allow with attribution and reviewer assignment |
| **T3 Restart/Modify/Deploy/Delete/Change Permissions** | restart workloads, apply Terraform, rotate secrets, alter RBAC, delete resources, production deploy | High-to-critical (service interruption, security exposure, irreversible actions) | Step-up, explicit approval, or deny based on env and policy |

The key principle is that **read-only diagnostics are not “less useful” tools**. They are valuable tools with lower mutation risk. A mature program prioritizes T0/T1 coverage first, then adds tightly controlled T2/T3 paths.

## 3) Enforcement model at the MCP tool-call layer

The enforcement model should evaluate every call using four inputs: **agent identity, delegating human, declared intent, and target environment**. This combination determines a trust ceiling. If a call exceeds this ceiling, policy will return one of three outcomes: allow, step-up, or deny.

First, classify tools by **operation risk**, not by server label. A “Microsoft Graph MCP” endpoint might offer both read operations and privilege-changing operations; they cannot share the same default trust treatment. The same applies to Azure MCP, where one method might only query Azure Monitor while another can modify production RBAC.

Second, bind delegated authority. If a low-trust assistant is delegated by an SRE, it should remain capped at T1 in production, even if the human could perform T3 actions directly. Delegation should not lead to implicit privilege escalation.

Third, separate **approval prompt vs policy step-up**. An approval prompt is a user interface interaction. A policy step-up is an authorization state transition requiring stronger proof (for example, specific approver role, fresh auth context, change window, and incident severity). The prompt is how humans interact; the policy is what actually governs.

```yaml
policy: mcp_tool_authorization
default: deny

rules:
  - when:
      env: ["dev","staging"]
      risk_class: ["inspect_read","diagnose_test","draft_change"]
      agent_trust: "low_or_higher"
    decision: allow

- when:
      env: ["prod"]
      risk_class: ["inspect_read","diagnose_test"]
      agent_trust: "low_or_higher"
    decision: allow

- when:
      env: ["prod"]
      risk_class: ["draft_change"]
      agent_trust: "medium_or_higher"
      delegating_human_role: ["SRE","PlatformEngineer","SecOps"]
    decision: allow

- when:
      env: ["prod"]
      risk_class: ["modify_deploy_delete","iam_permission_change","secret_rotation","terraform_apply","restart_service"]
      agent_trust: "any"
    decision: step_up
    requires:
      approvals: 1
      approver_role: ["IncidentCommander","ServiceOwner"]
      mfa_fresh_minutes: 15
      change_ticket: true

- when:
      env: ["prod"]
      risk_class: ["iam_permission_change","delete_resource"]
      incident_state: ["not_declared"]
    decision: deny
```

## 4) Audit chain model for AI ops actions

An AI ops audit trail should be an **investigation chain**, not just a chat transcript. At a minimum, it must preserve the evidence set the agent used, including source systems, query windows, and dataset versions.

It also needs exact tool-call records with parameters, normalized for review and replay. Without parameter-level visibility, you cannot distinguish “listed pods in namespace X” from “deleted deployment in namespace X.” The chain should then capture the hypothesis tree, with confidence deltas over time as each validation step succeeds or fails.

Every authorization decision must be logged with policy rationale: why the action was allowed, stepped up, or denied. If human-in-the-loop approval occurred, capture who approved, in what role, at what time, under which incident/change ID, and with what constraints. Finally, record the operational outcome and rollback notes so post-incident learning can connect decision quality to system impact.

## 5) Policy examples across enterprise tooling

For **Azure MCP and Azure Monitor**, treat KQL/log and metric reads as T0/T1, but any action that restarts services, alters autoscaling, or changes alerting in production should move to T3 controls. For **Key Vault**, secret metadata reads may be allowed for diagnostics, while secret rotation and access policy changes are high-trust operations requiring step-up or explicit approval.

For **Entra ID and RBAC** through Azure or Microsoft Graph MCP, role assignment changes, app consent, and directory permission grants are always high-risk. A low-trust incident bot can inspect sign-in anomalies and directory state, but permission changes should require an incident commander or service owner approval path.

For **GitHub**, read-only repository inspection and PR diff analysis can be broadly allowed, while direct pushes to protected branches, force merges, or secrets changes should require stronger controls. A medium-trust agent can open an issue or draft PR remediation plan without altering production behavior.

For **CI/CD**, allow pipeline log reads and artifact provenance checks by default, but block autonomous promotion to production unless step-up conditions are met. For **Kubernetes**, list/get/describe and event reads remain diagnostic; deletes, restarts, exec into privileged pods, and policy changes belong in T3. Across **cloud IAM** systems generally, principal creation, privilege grants, and trust-policy edits should be treated as critical operations with explicit human checkpoints.

## Where Permit fits as runtime enforcement and audit decision layer

This is where **Permit** fits: not as a generic chatbot feature, but as the runtime authorization brain for MCP/tool-call decisions and the decision log that links policy to action. In this architecture, the agent asks Permit for a decision on every planned tool call, including context like risk class, environment, delegating user, incident state, and requested operation.

That model makes the trust split operationally clear: a low-trust ops agent can list/read/inspect; a medium-trust agent can draft ticket/remediation plans; and high-trust actions such as IAM changes, secret rotation, production restart, deletion, or Terraform apply require policy step-up, explicit approval, or denial. It also standardizes approval semantics so “approve in chat” is not enough unless the policy engine confirms required role, freshness, and scope. The result is a defensible control plane for AI incident response that scales across Azure MCP, Microsoft Graph MCP, GitHub, CI/CD, Kubernetes, and IAM ecosystems.

## Frequently asked questions

### How should enterprises secure AI incident-response and ops agents?

Secure them as delegated actors with limited authority, not as autonomous admins. Start by separating diagnostic capability from mutation capability, then enforce trust ceilings at the MCP tool-call layer. Add human-in-the-loop approval for high-risk production changes and ensure every authorization decision is auditable.

### What is the difference between read-only diagnostics and remediation tools?

Read-only diagnostics gather and correlate evidence without changing system state, making them safer to run continuously in production. Remediation tools change infrastructure, identity permissions, or deployment state, and therefore carry outage and security risk. Both are valuable, but they belong to different trust tiers and approval paths.

### How do trust levels apply to MCP tools?

Trust levels apply to individual operations, not entire MCP servers. The same server can expose harmless reads and high-impact writes, so policy must classify each tool by risk class. Runtime authorization should evaluate agent trust, delegating human, intent, and environment before each tool call.

### What should an AI ops agent audit trail include?

It should include evidence gathered, exact tool calls and parameters, and a timestamped hypothesis tree with confidence updates. It must also record policy decisions with rationale and human approval metadata when step-up is required. Finally, it should capture outcome quality and rollback details for post-incident learning and compliance review.

### When should an agent need approval before changing infrastructure?

Approval is typically required for production actions that can impact availability, security posture, or data integrity. This includes IAM or RBAC changes, secret rotation, service restarts, deploys, deletes, and Terraform apply. In lower environments, you can relax controls, but production mutation should still be policy-gated with explicit step-up criteria.

### What is the difference between an approval prompt and a policy step-up?

An approval prompt is the interface moment where a human is asked to confirm something. A policy step-up is the authorization requirement that defines who can approve, under what conditions, and with what freshness or incident context. You need both, but governance fails if teams rely on prompts without enforceable policy checks.

### Written by

### Gabriel L. Manor

Full-Stack Software Technical Leader \| Security, JavaScript, DevRel, OPA \| Writer and Public Speaker

---