AI Security Incident Response

01 / SCOPE

Recognize AI incidents by violated boundaries

Escalate when AI-enabled behavior crosses a confidentiality, integrity, availability, safety, or authorization boundary. A surprising answer alone is not necessarily an incident; validate the behavior and its downstream effect.

Inputs and knowledge

Direct or indirect prompt injection
System or developer instruction manipulation
RAG, vector-store, memory, or knowledge-source poisoning
Sensitive-data disclosure through prompts, retrieval, or output

Agency and identity

Tool or function abuse
Excessive agency or unauthorized autonomous action
Delegated identity, session, authorization, or approval abuse
Model/API credentials, tokens, or secrets compromised

Behavior and supply chain

Malicious or unexpected model/tool behavior
Unauthorized model, prompt, agent, policy, or orchestration changes
Compromised models, packages, plugins, tools, connectors, or dependencies
Unexpected downstream writes, messages, transactions, deployments, or egress

02 / EVIDENCE

Preserve the control plane and execution path

A transcript alone is incomplete. Preserve raw records with UTC timestamps, correlation identifiers, source, collection method, configuration version, retention limits, and chain-of-custody metadata where required.

Minimum AI incident evidence by layer
Layer	Preserve	Questions
Conversation and instructions	Prompts, responses, system/developer instructions, safety policies, memory, templates, timestamps, correlation IDs	What influenced the model, and what did the user see?
Model and agent	Provider, model and version, agent configuration, orchestration graph, sampling/runtime settings, change history	Which executable configuration produced the behavior?
Tools and execution	Tool/function calls and parameters, tool results, approvals, denials, retries, agent execution traces, queue and job state	What action was attempted, authorized, and completed?
Retrieval and knowledge	RAG queries, retrieved documents and versions, rankings, index/vector-store changes, ingestion and source provenance	Was untrusted or modified knowledge used?
Identity and authorization	User, workload and agent identities; sessions; delegated grants; roles; policy decisions; human approvers	Whose authority was used and was it valid?
Provider and application	Model/API activity, data-access logs, network activity, application/security telemetry, content filters and policy evaluations	What data moved and which systems were affected?

Preserve ephemeral state early. Pause retention expiry where authorized and export queues, scratchpads, memory, retrieval results, temporary credentials, execution traces, provider request IDs, and in-flight approvals before stopping or redeploying components. Minimize unrelated personal or customer data.

03 / LIFECYCLE

Run an iterative AI incident lifecycle

ValidateReproduce safely, confirm boundary violation, distinguish model uncertainty from completed action.

ScopeMap identities, agents, models, tools, knowledge, data, destinations, and downstream effects.

ContainReduce unsafe authority while preserving evidence and essential service.

EradicateRemove poisoned sources, compromised credentials, unsafe logic, persistence, and vulnerable dependencies.

RecoverRollback to trusted versions, stage capability restoration, and validate target systems.

ImproveAdd telemetry, approval gates, tests, ownership, and verified corrective actions.

Human authority gate

Before a disruptive action, identify the approving role, affected business and safety owners, expected impact, evidence tradeoff, rollback path, success signal, and emergency authority. Prefer reversible capability reduction; act immediately when delay creates greater harm and record the decision.

AI-IR-01

AI / LLM incident

Triggers: confirmed or credible evidence of prompt injection, unsafe tool use, unauthorized agent action, retrieval poisoning, sensitive disclosure, identity or credential abuse, malicious model/tool behavior, unapproved configuration change, or AI supply-chain compromise.

First 15 minutes

Assign incident lead, AI/platform lead, scribe, severity, UTC start time, and protected channel.
Validate the boundary violation and determine whether unsafe execution, data access, or egress remains active.
Pause new agent runs or queues when justified; restrict high-impact and write-capable tools without deleting state.
Snapshot instructions, model/version, agent and policy configuration, memory, retrievals, traces, tool transcripts, identities, credentials metadata, and target logs.
Identify immediate human-safety, financial, legal, privacy, production, and business-continuity constraints.

15–60 minutes

Build the UTC timeline from input through orchestration, retrieval, model output, authorization, tool execution, and downstream result.
Map blast radius across users, tenants, agents, identities, reachable tools, data sources, destinations, scheduled work, and prior executions.
Verify actual reads, writes, messages, transactions, deployments, and egress in authoritative target-system logs.
Quarantine poisoned sources or compromised connectors and revoke narrowly scoped affected sessions, tokens, or API keys when exposure justifies it.
Choose the last trusted model, prompt, policy, agent, tool allowlist, retrieval index, and orchestration configuration; define recovery tests and approvers.

Authority and decision gates

Stop/pause gate: Is harmful agency active, and will stopping it affect safety, critical operations, evidence, or recovery?
Revocation gate: Is the token, session, API key, service principal, or delegated grant exposed or abused? Revoke the narrowest affected authority and plan replacement dependencies.
Isolation gate: Can the RAG source, index, plugin, model endpoint, or tool be quarantined without losing the only evidence copy?
Rollback gate: Is the proposed version known-good, compatible, scanned, reproducible, and reversible?
Communication gate: Do facts support customer, provider, regulator, or partner notification, and who is authorized to communicate?

Exit criteria

Unsafe execution and unauthorized data paths are stopped and independently verified.
Affected identities, secrets, sessions, agents, tools, models, knowledge sources, and downstream systems are scoped to documented confidence.
Poisoning, unsafe configuration, vulnerable dependencies, and persistence are removed from a trusted recovery point.
Recovery tests cover adversarial prompts, indirect content, authorization, tool boundaries, data access, rollback, logging, and business transactions.
Security and service owners approve staged restoration, monitoring duration, residual risk, rollback triggers, and accountable operator.

04 / CONTAINMENT OPTIONS

Reduce capability proportionally

Containment choices and safeguards
Action	Use when	Safeguard and validation
Pause agents, schedules, or queues	Unsafe autonomous execution remains possible	Preserve in-flight state; verify no new runs; provide a manual continuity path
Remove write/high-impact tools	Read-only service can safely continue	Use an allowlist and independent policy enforcement; test denied actions
Require human approval	Risk can be bounded by accountable review	Approver must see intent, parameters, target, data, and expected effect; prevent self-approval
Revoke sessions, tokens, or API keys	Authority is exposed, replayed, or abused	Map dependencies, preserve metadata, rotate consumers, and verify old material fails
Isolate RAG/data sources	Content or ingestion path is poisoned or unauthorized	Snapshot source and index; block ingestion/retrieval; validate clean provenance
Rollback model/configuration	A change introduced unsafe behavior	Use signed/versioned known-good artifacts, canaries, kill switch, and enhanced monitoring

05 / RECOVERY & IMPROVEMENT

Restore authority in stages

Restore observation and read-only paths first; then low-impact writes; then privileged or irreversible actions.
Test expected tasks plus direct and indirect prompt injection, confused-deputy paths, cross-tenant access, poisoned retrieval, tool parameter tampering, and approval bypass.
Compare model, prompt, policy, agent, orchestration, dependency, plugin, tool, and knowledge versions to the approved baseline.
Validate target-system records—not only agent output—and monitor provider/API, identity, network, data-access, and application telemetry.
Document root cause, affected parties, evidence limitations, containment impact, corrective owners, due dates, and verification method.

Lessons learned: convert findings into versioned adversarial tests, least-privilege tool scopes, independent authorization, provenance controls, retention changes, rollback drills, and tabletop scenarios. Close actions only after evidence shows the control works.

Download the AI incident evidence checklist