Automated Key Rotation and Emergency Key Compromise Response

The Assumption

Key rotation is a rare event triggered by incident response. The assumption: keys do not need regular rotation if they have not been compromised.

PCI DSS requires annual cryptographic key rotation regardless of compromise. SOC 2 auditors ask for rotation schedules. NIST SP 800-57 recommends rotation periods based on key type and usage volume. But compliance is not the primary reason to rotate keys regularly. The primary reason: a key that has been in use for 3 years has been stored in 3 years of backups, memory dumps, configuration snapshots, and deployment artifacts. The longer a key lives, the larger the attack surface.

Regular rotation also exercises the rotation process. Teams that rotate keys annually discover on rotation day that their process is broken. Teams that rotate keys weekly discover that on the first week and fix it.

The Attack

Scenario: Key extracted from old backup. The SaaS platform’s signing key was generated 2 years ago. In those 2 years, the key has been backed up (encrypted, but the backup encryption key was also backed up somewhere), included in Docker images (baked into the container as an environment variable), and present in memory dumps from incident investigations.

An attacker gains access to a 6-month-old backup tape. The backup contains the signing key. Because the key has never been rotated, it is still the active signing key. The attacker forges tokens at will.

If the key had been rotated every 90 days, the key from the 6-month-old backup would have been retired 3 months ago and would no longer be in the JWKS. Forged tokens signed with the old key would be rejected.

The Spec or Mechanism

Key Lifecycle States

The state machine enforces a strict progression. A key starts as PENDING (generated but not published). It moves to ACTIVE (published in JWKS for validation caching) before becoming the SIGNING key. After rotation, it enters RETIRING (still in JWKS for validating in-flight tokens) before removal from JWKS as RETIRED, and finally DESTROYED when the cryptographic material is wiped. The critical insight is that ACTIVE and RETIRING states exist to give clients time to cache new keys and finish validating tokens signed with old keys.

State	In JWKS?	Used for signing?	Used for validation?
PENDING	No	No	No
ACTIVE	Yes	No	Yes
SIGNING	Yes	Yes	Yes
RETIRING	Yes	No	Yes
RETIRED	No	No	No
DESTROYED	No	No	No

Only one key should be in SIGNING state at any time. Multiple keys can be in ACTIVE or RETIRING state simultaneously. The transition from RETIRING to RETIRED should happen after the maximum token TTL has elapsed (no unexpired tokens signed with the retiring key can exist).

The Implementation

Key Entity and Repository

@Entity
@Table(name = "signing_keys")
public class SigningKeyEntity {

    @Id
    private String keyId;

    @Enumerated(EnumType.STRING)
    private KeyStatus status;

    @Column(columnDefinition = "TEXT")
    private String privateKeyPem;

    @Column(columnDefinition = "TEXT")
    private String publicKeyPem;

    private Instant createdAt;
    private Instant activatedAt;
    private Instant signingStartedAt;
    private Instant retiringStartedAt;
    private Instant retiredAt;
    private Instant destroyedAt;
    private String retirementReason;

    // The private key PEM should be encrypted at rest.
    // Use Spring Vault or AWS KMS envelope encryption in production.
}

public enum KeyStatus {
    PENDING, ACTIVE, SIGNING, RETIRING, RETIRED, DESTROYED
}

Automated Rotation Scheduler

// VULNERABLE: Manual rotation, no schedule
// The key was generated in 2022 and never rotated.
// No one remembers how to rotate it.
// The runbook was written by someone who left the company.

// HARDENED: Automated rotation on schedule
@Component
public class KeyRotationScheduler {

    private final KeyRotationService keyRotationService;
    private final Duration rotationInterval = Duration.ofDays(90);
    private final Duration retirementGracePeriod = Duration.ofHours(24);

    /**
     * Daily check: is the current signing key older than the rotation interval?
     */
    @Scheduled(cron = "0 0 3 * * *") // 3 AM daily
    public void checkRotationSchedule() {
        SigningKeyEntity currentKey = keyRotationService.getCurrentSigningKey();

        if (currentKey == null) {
            // No signing key exists. This is a bootstrap scenario.
            keyRotationService.bootstrapFirstKey();
            return;
        }

        Duration keyAge = Duration.between(currentKey.getSigningStartedAt(), Instant.now());

        if (keyAge.compareTo(rotationInterval) > 0) {
            performScheduledRotation(currentKey);
        }

        // Also retire any keys that have been in RETIRING state
        // longer than the grace period
        retireOldKeys();
    }

    private void performScheduledRotation(SigningKeyEntity currentKey) {
        // Phase 1: Introduce new key (ACTIVE state, in JWKS, not signing)
        String newKeyId = keyRotationService.generateKey();
        keyRotationService.activateKey(newKeyId);

        // Phase 2: Promote new key to signing, demote old key to retiring
        keyRotationService.promoteToSigning(newKeyId);
        keyRotationService.demoteToRetiring(currentKey.getKeyId(), "scheduled rotation");

        // Phase 3 happens in retireOldKeys() after grace period
    }

    private void retireOldKeys() {
        List<SigningKeyEntity> retiringKeys = keyRotationService.getKeysByStatus(
            KeyStatus.RETIRING);

        for (SigningKeyEntity key : retiringKeys) {
            Duration retiringDuration = Duration.between(
                key.getRetiringStartedAt(), Instant.now());

            if (retiringDuration.compareTo(retirementGracePeriod) > 0) {
                keyRotationService.retireKey(key.getKeyId());
            }
        }

        // Destroy keys that have been retired for 30 days
        List<SigningKeyEntity> retiredKeys = keyRotationService.getKeysByStatus(
            KeyStatus.RETIRED);

        for (SigningKeyEntity key : retiredKeys) {
            Duration retiredDuration = Duration.between(
                key.getRetiredAt(), Instant.now());

            if (retiredDuration.compareTo(Duration.ofDays(30)) > 0) {
                keyRotationService.destroyKey(key.getKeyId());
            }
        }
    }
}

Key Audit Trail

@Entity
@Table(name = "key_rotation_audit")
public class KeyRotationAudit {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    private String keyId;
    private String previousStatus;
    private String newStatus;
    private String reason;
    private String performedBy; // "scheduler" or admin username
    private Instant performedAt;
    private String metadata; // JSON: rotation results, affected services
}

@Service
public class KeyRotationService {

    private final SigningKeyRepository keyRepository;
    private final KeyRotationAuditRepository auditRepository;

    @Transactional
    public void promoteToSigning(String keyId) {
        // Demote current signing key
        keyRepository.findByStatus(KeyStatus.SIGNING).ifPresent(current -> {
            KeyStatus previous = current.getStatus();
            current.setStatus(KeyStatus.ACTIVE);
            keyRepository.save(current);
            logAudit(current.getKeyId(), previous, KeyStatus.ACTIVE,
                "demoted: new signing key promoted");
        });

        // Promote new key
        SigningKeyEntity key = keyRepository.findById(keyId)
            .orElseThrow(() -> new IllegalArgumentException("Key not found: " + keyId));

        KeyStatus previous = key.getStatus();
        key.setStatus(KeyStatus.SIGNING);
        key.setSigningStartedAt(Instant.now());
        keyRepository.save(key);

        logAudit(keyId, previous, KeyStatus.SIGNING, "promoted to signing");
    }

    private void logAudit(String keyId, KeyStatus previous,
            KeyStatus newStatus, String reason) {
        KeyRotationAudit audit = new KeyRotationAudit();
        audit.setKeyId(keyId);
        audit.setPreviousStatus(previous.name());
        audit.setNewStatus(newStatus.name());
        audit.setReason(reason);
        audit.setPerformedBy("scheduler");
        audit.setPerformedAt(Instant.now());
        auditRepository.save(audit);
    }
}

The Verification

@SpringBootTest
class AutomatedKeyRotationTest {

    @Autowired
    private KeyRotationService keyRotationService;

    @Autowired
    private KeyRotationScheduler scheduler;

    @Autowired
    private SigningKeyRepository keyRepository;

    @Autowired
    private KeyRotationAuditRepository auditRepository;

    @Test
    void scheduledRotationCreatesNewSigningKey() {
        // Bootstrap initial key
        keyRotationService.bootstrapFirstKey();
        SigningKeyEntity originalKey = keyRotationService.getCurrentSigningKey();
        String originalKeyId = originalKey.getKeyId();

        // Backdate the key to trigger rotation
        originalKey.setSigningStartedAt(Instant.now().minus(Duration.ofDays(91)));
        keyRepository.save(originalKey);

        // Run scheduler
        scheduler.checkRotationSchedule();

        // Verify: new signing key exists
        SigningKeyEntity newKey = keyRotationService.getCurrentSigningKey();
        assertThat(newKey.getKeyId()).isNotEqualTo(originalKeyId);
        assertThat(newKey.getStatus()).isEqualTo(KeyStatus.SIGNING);

        // Verify: old key is in RETIRING state (still in JWKS)
        SigningKeyEntity oldKey = keyRepository.findById(originalKeyId).orElseThrow();
        assertThat(oldKey.getStatus()).isEqualTo(KeyStatus.RETIRING);

        // Verify: audit trail records the rotation
        List<KeyRotationAudit> audits = auditRepository.findByKeyId(originalKeyId);
        assertThat(audits).isNotEmpty();
        assertThat(audits.stream().map(KeyRotationAudit::getNewStatus))
            .contains("RETIRING");
    }

    @Test
    void retiringKeyRemovedAfterGracePeriod() {
        keyRotationService.bootstrapFirstKey();
        SigningKeyEntity key = keyRotationService.getCurrentSigningKey();

        // Simulate: key was moved to RETIRING 25 hours ago
        key.setStatus(KeyStatus.RETIRING);
        key.setRetiringStartedAt(Instant.now().minus(Duration.ofHours(25)));
        keyRepository.save(key);

        // Create a current signing key
        keyRotationService.bootstrapFirstKey();

        // Run scheduler
        scheduler.checkRotationSchedule();

        // Old key should be RETIRED (no longer in JWKS)
        SigningKeyEntity retiredKey = keyRepository.findById(key.getKeyId()).orElseThrow();
        assertThat(retiredKey.getStatus()).isEqualTo(KeyStatus.RETIRED);
    }

    @Test
    void onlyOneSigningKeyAtAnyTime() {
        keyRotationService.bootstrapFirstKey();

        // Trigger rotation multiple times
        for (int i = 0; i < 3; i++) {
            SigningKeyEntity current = keyRotationService.getCurrentSigningKey();
            current.setSigningStartedAt(Instant.now().minus(Duration.ofDays(91)));
            keyRepository.save(current);
            scheduler.checkRotationSchedule();
        }

        // Verify: exactly one SIGNING key
        List<SigningKeyEntity> signingKeys = keyRepository.findAllByStatus(KeyStatus.SIGNING);
        assertThat(signingKeys).hasSize(1);
    }
}

The third test validates the invariant: at any point in time, exactly one key is in SIGNING state. Multiple signing keys would mean tokens are signed with unpredictable keys, making debugging impossible and invalidation unreliable.

Regular rotation limits blast radius. A key that has been signing tokens for three months has three months of outstanding tokens that become suspect if the key is later found to be compromised. A key that rotates weekly has one week of suspect tokens. The operational cost of rotation must be low enough to make frequent rotation painless. That means automation.