An agent incident is not simply a bad model output. It is an event in which an AI-enabled workflow creates, attempts, or enables an action that violates an expected security property. That could mean disclosing retrieved data, using a tool outside intended scope, following injected instructions, or causing a downstream system change.
1. Recognize the incident
Treat behavior as an incident when it crosses an authorization, confidentiality, integrity, or availability boundary—not merely when the output is surprising. Start with observable facts.
- A tool call was attempted or completed outside the user's intended task.
- Retrieved content appears to have influenced higher-priority instructions.
- Sensitive context reached an output, log, external endpoint, or unintended tenant.
- An agent identity exercised a permission its owner did not understand it possessed.
2. Establish scope across four planes
| PLANE | QUESTIONS | EVIDENCE |
|---|---|---|
| Identity | Which user, service account, token, or delegated credential acted? | Authentication and token-exchange logs |
| Memory | What conversation, state, cache, or vector-store context persisted? | Session state, retrieval traces, embeddings metadata |
| Tools | What could the agent read or change, and what actually executed? | Tool gateway, API, and target-system audit logs |
| Instructions | Which system, developer, user, or retrieved instructions shaped the action? | Versioned prompts and policy decisions |
3. Contain without destroying the trail
- Pause new runs. Disable the workflow entry point or queue, not the underlying evidence stores.
- Revoke delegated credentials. Prefer the narrowest affected agent, integration, or session before tenant-wide revocation.
- Block risky tools. Move write-capable tools to deny-by-default while keeping read-only telemetry available.
- Snapshot state. Preserve configuration, prompt versions, memory, retrieval results, policy output, and tool transcripts.
- Hunt downstream. Verify every system the agent could reach, not only those visible in the final response.
Do not trust the conversation transcript alone
User-visible messages frequently omit tool arguments, retries, policy evaluations, retrieved chunks, and intermediate model output. The authoritative timeline is assembled from control-plane and target-system logs.
4. Preserve the minimum viable evidence set
Record immutable copies of the agent definition, model and parameter versions, effective instructions, full tool-call envelopes, identity claims, retrieval identifiers, timestamps, policy decisions, and human approvals. Hash exports when chain of custody matters.
Avoid collecting unrelated prompt or user data. Evidence preservation still needs purpose limitation, access control, and a retention decision.
5. Recover with confidence
Recovery is not “turn it back on.” Reproduce the behavior in an isolated environment, add a control that breaks the demonstrated attack path, and test both the exploit and legitimate workflows. Restore capability in stages: read-only tools, low-impact writes, then privileged actions with explicit approval.
Field checklist
- Assign an incident commander and record UTC timestamps.
- Freeze prompt, policy, model, tool, and connector versions.
- Map the agent's effective identity and all delegated credentials.
- List reachable tools and validate actual downstream actions.
- Preserve retrieval and memory references without over-collecting content.
- Test containment against legitimate and adversarial cases.
- Document residual risk and require an owner to accept it.