Prompt injection & tool abuse
Untrusted input overrides intended behavior or drives a tool beyond the user's authorized task.
OPERATIONAL CAPABILITY · AI-IR
AI Security Incident Response
A field framework for incidents where models, agents, retrieval systems, identities, and tools can turn a bad instruction into a real downstream action.
What counts as an AI incident
Escalate when AI-enabled behavior crosses a confidentiality, integrity, availability, safety, or authorization boundary—not merely because an answer is surprising.
Sensitive-data disclosure
Prompts, retrieval, memory, logs, or model output expose protected data to the wrong identity or destination.
Agent compromise
An agent identity, runtime, memory, connector, or delegated credential is controlled or persistently influenced.
Poisoned knowledge sources
A retrieval source, embedding pipeline, index, or cached artifact is manipulated to influence decisions.
Model / API credential compromise
A model-provider key, service token, or delegated credential is exposed or used outside its expected context.
Excessive agency
The system can plan or act with broader tools, permissions, duration, or autonomy than the approved risk boundary.
Malicious model or tool behavior
A model, plugin, MCP server, dependency, or downstream tool returns deceptive output or performs hidden actions.
Telemetry and evidence
Preserve the control-plane and target-system record. A conversation transcript alone omits tool arguments, retries, policy evaluations, retrievals, and hidden downstream effects.
Inputs
Prompts, responses, system/developer instructions, attachments, and safety or policy output.
Runtime
Model/provider and version, parameters, agent definition, session state, memory, and release identifiers.
Actions
Tool calls and results, agent traces, retries, approvals, queue events, and target-system audit records.
Retrieval
Queries, retrieved chunks, source/version identifiers, ranking, filters, embeddings metadata, and index changes.
Identity
Users, service principals, delegated credentials, token exchanges, authorization decisions, roles, and policy versions.
Data access
Model/API activity, connector access, read/write events, egress destinations, and relevant application or cloud logs.
Evidence rule: export immutably, record collection time and source, hash when chain of custody matters, restrict access, and set a purpose-based retention decision. Avoid collecting unrelated user content.
Investigate → contain → eradicate → recover
The phases are iterative. New indicators can send the team back to containment. Every transition has an explicit decision gate and a documented owner.
Investigate
Freeze versions and build a UTC timeline across identity, instructions, memory, retrieval, tools, and downstream systems.
Decision gate. Affected identities, actions, data, destinations, and earliest known influence are bounded—or uncertainty is explicitly accepted.
Contain
Pause new runs; deny write-capable tools; revoke the narrowest affected credentials; quarantine poisoned sources; preserve telemetry.
Decision gate. Unsafe actions and data egress are stopped, evidence remains available, and business impact is understood.
Eradicate
Remove injected or poisoned content, rebuild compromised components, rotate exposed secrets, and close the failed authorization path.
Decision gate. The demonstrated attack path no longer succeeds in isolation and no persistence mechanism remains.
Recover
Restore read-only capability first, then low-impact writes, then privileged actions with explicit approval and enhanced monitoring.
Decision gate. Security and service owners approve residual risk, rollback triggers, monitoring window, and accountable operator.
Rollback safeguard: define the last known-good model, prompt, policy, retrieval index, tool allowlist, and credentials before recovery. Use canaries and kill switches; never roll forward if telemetry is incomplete.
AI-IR-01 · FIRST 15 MINUTES
AI / LLM incident playbook
- Declare and own. Assign an incident commander, severity, scribe, and UTC start time.
- Stop unsafe agency. Pause new runs or queues and deny write-capable or high-impact tools. Do not delete prompts, memory, traces, or indexes.
- Preserve active state. Snapshot instructions, model/version, agent configuration, policy, retrievals, memory, tool transcripts, identities, and target logs.
- Bound identity and reach. Map users, service principals, delegated tokens, reachable tools, data stores, and external destinations.
- Revoke and quarantine. Rotate potentially exposed model/API credentials and isolate poisoned sources, compromised agents, connectors, or tools.
- Verify downstream actions. Confirm actual reads, writes, messages, transactions, deployments, and egress in authoritative target logs.
- Set the next gate. Name the evidence needed before eradication, rollback point, recovery test, and approver.
AUTHORSHIP & REVIEW
How to trust and adapt this guide
- Author & accountable editor
- Leandro Rocha
- Version
- 1.0
- Published / reviewed
- 18 August 2026
- Review status
- Maintainer-reviewed; independent peer review is not claimed.
- Editorial process
- Read the sourcing, corrections, and AI-assistance policy ↗
Reference basis
- NIST SP 800-61 Rev. 3
- NIST AI Risk Management Framework
- OWASP Top 10 for LLM Applications
- MITRE ATLAS
Assumptions
- Model, prompt, agent, retrieval, identity, policy, and tool telemetry were enabled before the incident.
- The organization can pause agent execution and revoke delegated credentials without deleting evidence.
Limitations
- Model behavior is probabilistic; reproducing one output does not prove the full incident scope.
- Provider telemetry, retention, model-version visibility, and tool traces vary by platform and contract.