Incident response for compromised machine identities

An API key appears in a public repository. A service principal authenticates from an unfamiliar environment. A Kubernetes workload starts requesting tokens for resources it has never used.

These can look like simple credential incidents, but rotating one secret is rarely sufficient.

A machine identity is a principal used by software rather than a person. Its authority may involve a service account, workload or managed identity, API key, OAuth application, certificate, cloud role session, or delegated token.

Deleting a credential does not necessarily invalidate tokens already issued. Revoking a token does not remove a malicious certificate added to the principal. Disabling a workload does not eliminate a second identity created for persistence.

The response objective is not merely to replace a secret. It is to stop the attacker from authenticating, minting new authority, reaching protected resources, delegating access, or rebuilding persistence.

Recognizing machine-identity abuse

Traditional user detections—impossible travel, suspicious MFA prompts, or interactive sign-ins—may provide little value when the principal is software. Look instead for changes in how the identity is created, authenticated, authorized, and used.

Identity and credential signals

• A new password, certificate, federated credential, or API key.
• Credential creation outside the approved deployment pipeline.
• A certificate thumbprint absent from the configuration baseline.
• Authentication from an unexpected network, cloud, region, or workload.
• Token requests with an unfamiliar issuer, audience, scope, or subject.
• Tokens minted after the expected workload was stopped.
• A sudden increase in role assumptions or token exchanges.
• Changes to application owners, redirect URIs, permissions, or consent.
• A disabled identity being re-enabled.
• A replacement identity with a similar display name or purpose.

Microsoft recommends examining service-principal sign-ins, credential changes, app-role assignments, consent grants, redirect URIs, custom roles, and affected resources when investigating compromised applications. Microsoft Security Operations ↗

Resource-plane signals

• Access to a new subscription, account, vault, database, or tenant.
• Resource enumeration followed by bulk reads.
• Secret retrieval outside the normal deployment window.
• Privilege or policy modifications.
• New automation jobs, functions, queues, webhooks, or scheduled tasks.
• Requests using a retired key identifier.
• Administrative changes made through an unexpected API or tool.
• Unusual access volume from an otherwise legitimate workload.

Start with an identity graph

Before changing the environment, map the principal and every path through which it can obtain, exercise, or delegate authority.

• Immutable principal and application identifiers.
• Tenant, account, subscription, cluster, and namespace.
• Credential identifiers, creation dates, and expiration dates.
• Certificate thumbprints and key identifiers—never private material.
• Workloads and deployment pipelines using the identity.
• Direct and inherited roles, group memberships, permissions, and scopes.
• Consent grants and resource-based policies.
• Impersonation and token-creation privileges.
• Vaults and secret stores accessible to the identity.
• Federated issuers, audiences, subjects, and trust rules.
• Active sessions and maximum token lifetimes.
• Resources contacted before and after the first suspicious event.

Identity-containment architecture

Containment must operate at multiple control points. No single identity-provider action is reliable enough for every platform and every previously issued token.

Responders may need to disable the principal, remove its federation rule, revoke active tokens, apply an explicit deny at the resource plane, stop the workload, and only then rotate its credentials.

The containment sequence

0–15 minutes: preserve identifiers and stop new authority

Capture principal and application IDs, credential IDs, thumbprints, roles, groups, consent grants, federation configuration, recent token events, source workload, deployment pipeline, and relevant resource activity before deleting anything.

Then prevent the principal from minting new authority: disable the account or service principal, remove or narrow workload federation, suspend the OAuth application, detach the managed identity, stop the workload, or block the identity at the gateway.

Microsoft recommends disabling sign-in for a compromised application to create investigation time before removing the object or replacing every credential. Microsoft Security Operations ↗

15–30 minutes: deny access at the resource plane

Assume previously issued tokens may continue working. Apply explicit deny policies, resource-policy changes, gateway restrictions, vault access removal, private-endpoint controls, account quarantine, or database login disablement.

AWS notes that temporary credentials usually remain valid until they expire unless their permissions are removed or revoked. Policy changes may also take several minutes to propagate. AWS IAM documentation ↗

30–60 minutes: revoke derived authority

Invalidate access and refresh tokens, cloud role sessions, application sessions, OAuth grants, delegated permissions, cached gateway credentials, impersonation tokens, and downstream SaaS sessions.

OAuth revocation can invalidate a token and, depending on the authorization server, related tokens or the underlying grant. Propagation delays and unsupported access-token revocation can leave residual access until expiration. RFC 7009 ↗

Verify revocation at the resource server. A successful identity-provider response does not prove that every downstream service has stopped accepting the token.

1–4 hours: rotate and redeploy

1. Create the replacement credential through an approved administrative path.
2. Deliver it through the managed secret store—not email, chat, or an incident ticket.
3. Update one consumer or workload group at a time.
4. Record which key identifier each consumer uses.
5. Validate expected authentication and resource access.
6. Disable and then remove the old credential.
7. Search for continued use of the retired identifier.
8. Redeploy affected workloads from trusted artifacts.

Immediately rotating every related secret can create a widespread outage, obscure which credential was used, and leave an attacker-controlled token or secondary credential untouched.

Containment by identity type

Managed identities and federated workloads require special attention because there may be no long-lived secret to rotate. The containment target is the trust relationship, role assignment, workload, or token issuer.

Google recommends short-lived service-account credentials and workload identity federation over user-managed service-account keys. Some services may not record which identity created a short-lived credential, making surrounding workload and resource logs important. Google Cloud IAM documentation ↗

Hunt for persistence, not only activity

Identity configuration

• Additional passwords, API keys, certificates, or federated credentials.
• Recently created or lookalike service accounts and applications.
• New application owners, roles, group memberships, or consent grants.
• Modified redirect URIs and custom roles built around the identity.

Federation and delegation

• Added OIDC issuers or broadened subject and audience patterns.
• Repository or branch wildcards in CI/CD trust policies.
• Token-creator and impersonation permissions.
• Delegation chains, cross-account trust, and new verification keys.

Workloads and resource policies

• Scheduled functions, workflows, containers, and automation rules.
• Modified manifests, infrastructure-as-code, pipeline variables, or startup scripts.
• Credentials copied into backups, logs, artifacts, or support bundles.
• Vault, storage, database, messaging, SaaS, and gateway policies granting access independently of central IAM.

Persistence frequently survives because responders clean the identity provider but do not inspect the resource policy—or clean the resource policy while leaving the token issuer compromised.

Preserve evidence without preserving the secret

Correlate immutable principal and application IDs, credential key IDs, certificate thumbprints, issuer, subject, audience, token or session ID, scopes, roles, creation and expiration times, source workload, deployment ID, request ID, target resource, action, and result.

principal_id · application_id · credential_key_id
token_issuer · token_subject · token_audience · session_id
issued_at · expires_at · source_ip · workload_instance
deployment_id · request_id · target_resource · action · result

Do not place passwords, private keys, API keys, bearer tokens, or reusable session cookies in case notes. When correlation requires a sensitive value, record a cryptographic fingerprint or the provider’s non-secret credential identifier.

Normalize identity-provider, cloud, workload, gateway, and target-system events to UTC while retaining each source’s original timestamp and ingestion time.

Recovery should be staged

1. Denied: The identity cannot authenticate or reach resources.
2. Canary: A replacement receives minimal read-only access in a controlled workload.
3. Limited: Required production permissions return for a small workload set.
4. Operational: Normal capacity returns with enhanced monitoring.
5. Hardened: Temporary controls become durable engineering and policy changes.

Before full recovery, confirm the administrative and deployment paths are trusted; every credential and trust path has an owner; unauthorized credentials, grants, and roles are removed; sessions are revoked or expired; workloads were rebuilt; resource policies were reviewed; retired credential use has stopped; high-risk activity was investigated; and a rollback method exists.

Engineering improvements after the incident

• Use workload federation or managed identities instead of static keys.
• Prefer short-lived, audience-bound credentials.
• Separate identities by workload and environment.
• Apply minimal roles with resource-level restrictions.
• Automate credential inventory and ownership.
• Alert on credential, consent, federation, and role changes.
• Centralize identity and resource-plane logging.
• Record non-secret key identifiers in authentication telemetry.
• Test emergency deny controls before an incident.
• Exercise rotation procedures against real dependency maps.
• Enforce trust and role configuration through infrastructure-as-code.
• Prevent secrets from entering repositories, logs, and build artifacts.

Short-lived credentials limit exposure, but they do not eliminate the need for revocation, resource enforcement, or logging. If an attacker can continuously request new tokens, a one-hour lifetime provides little protection.

Responder checklist

• Identify the principal, credentials, sessions, and trust relationships.
• Preserve credential identifiers, role assignments, grants, and recent logs.
• Stop new token or credential issuance.
• Apply resource-plane denial and revoke delegated authority.
• Stop or isolate the originating workload.
• Enumerate reachable resources and privilege-escalation paths.
• Extend the investigation through the maximum token lifetime.
• Remove unauthorized credentials, consent, roles, federation, and resource policies.
• Rebuild affected workloads from trusted artifacts.
• Restore access in stages and confirm retired credentials are no longer used.

Closing perspective

Human-account incidents are often organized around the user’s session. Machine-identity incidents have a wider shape: the principal, workload, credential, token issuer, trust policy, and every resource that accepts the resulting authority.

That is why “rotate the key” is not an incident-response strategy.