Fallback Testing and Validation

A fallback that has never executed in a test is a hypothesis, not a feature.

Testing Fraud Fallback Behavior

// PRODUCTION - Integration test for fraud fallback
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@Testcontainers
class FraudFallbackTest {

    @Container
    static GenericContainer<?> wireMock = new GenericContainer<>(
            DockerImageName.parse("wiremock/wiremock:latest"))
            .withExposedPorts(8080);

    @DynamicPropertySource
    static void configureProperties(DynamicPropertyRegistry registry) {
        registry.add("fraud.service.url", () ->
                "http://localhost:" + wireMock.getMappedPort(8080));
    }

    @Autowired
    private TestRestTemplate restTemplate;

    @Autowired
    private MeterRegistry meterRegistry;

    @Test
    void lowValuePayment_fraudServiceDown_autoApproved() {
        // Fraud service returns connection refused (no WireMock stub)
        stubFraudServiceDown();

        PaymentRequest request = new PaymentRequest(
                "user-1", new BigDecimal("25.00"), "USD");

        ResponseEntity<PaymentResult> response = restTemplate.postForEntity(
                "/api/payments", request, PaymentResult.class);

        assertThat(response.getStatusCode()).isEqualTo(HttpStatus.OK);
        assertThat(response.getBody().fraudScore().approved()).isTrue();
        assertThat(response.getBody().fraudScore().flaggedForReview()).isTrue();

        // Verify fallback metrics
        double fallbackCount = meterRegistry.counter(
                "fraud.fallback.invoked").count();
        assertThat(fallbackCount).isGreaterThan(0);

        double autoApprovedCount = meterRegistry.counter(
                "fraud.fallback.auto_approved").count();
        assertThat(autoApprovedCount).isGreaterThan(0);
    }

    @Test
    void highValuePayment_fraudServiceDown_rejected() {
        stubFraudServiceDown();

        PaymentRequest request = new PaymentRequest(
                "user-1", new BigDecimal("500.00"), "USD");

        ResponseEntity<PaymentResult> response = restTemplate.postForEntity(
                "/api/payments", request, PaymentResult.class);

        // High-value payment should be rejected when fraud service is down
        assertThat(response.getStatusCode()).isEqualTo(HttpStatus.OK);
        assertThat(response.getBody().status()).isEqualTo("REJECTED");

        double rejectedCount = meterRegistry.counter(
                "fraud.fallback.rejected").count();
        assertThat(rejectedCount).isGreaterThan(0);
    }

    @Test
    void multipleServicesDown_composedFallbackBehavior() {
        // Both fraud and notification services are down
        stubFraudServiceDown();
        stubNotificationServiceDown();

        PaymentRequest request = new PaymentRequest(
                "user-1", new BigDecimal("25.00"), "USD");

        ResponseEntity<PaymentResult> response = restTemplate.postForEntity(
                "/api/payments", request, PaymentResult.class);

        // Payment should succeed (fraud fallback auto-approves, notification queued)
        assertThat(response.getStatusCode()).isEqualTo(HttpStatus.OK);
        assertThat(response.getBody().status()).isEqualTo("SUCCESS");

        // Both fallbacks should have fired
        assertThat(meterRegistry.counter("fraud.fallback.invoked").count())
                .isGreaterThan(0);
        assertThat(meterRegistry.counter("notification.fallback.queued").count())
                .isGreaterThan(0);
    }

    private void stubFraudServiceDown() {
        String wireMockUrl = "http://localhost:" + wireMock.getMappedPort(8080);
        new RestTemplate().postForEntity(
                wireMockUrl + "/__admin/mappings",
                Map.of(
                        "request", Map.of("method", "POST", "url", "/api/fraud/score"),
                        "response", Map.of("status", 503, "body", "Service Unavailable")
                ),
                String.class
        );
    }

    private void stubNotificationServiceDown() {
        String wireMockUrl = "http://localhost:" + wireMock.getMappedPort(8080);
        new RestTemplate().postForEntity(
                wireMockUrl + "/__admin/mappings",
                Map.of(
                        "request", Map.of("method", "POST", "url", "/api/notifications"),
                        "response", Map.of("fault", "CONNECTION_RESET_BY_PEER")
                ),
                String.class
        );
    }
}

The third test is the critical one. It simulates two dependencies failing simultaneously and verifies that the composed fallback behavior is correct: the fraud fallback auto-approves (low value), the notification fallback queues for later, and the payment succeeds. This test catches the subtle bugs that appear only when multiple fallbacks compose: shared state, incorrect exception handling, and fallback methods that throw exceptions themselves.

Fallback Monitoring Dashboard

The fallback metrics feed a Grafana panel:

# Prometheus query: fallback invocation rate per dependency
sum(rate(fraud_fallback_invoked_total[5m])) by (reason)
sum(rate(notification_fallback_queued_total[5m]))
sum(rate(balance_fallback_payment_rejected_total[5m]))

The dashboard shows three panels, one per dependency with fallback capability. A spike in any fallback counter indicates a dependency is degraded. A sustained elevation indicates a prolonged outage. The reason label on the fraud counter distinguishes between timeouts (transient, likely to recover) and connection refused (persistent, may require intervention).

Alert when fraud fallback rate exceeds 10% of total fraud check rate for more than 5 minutes. This means more than 10% of payments are being processed without fraud scoring, which is a risk exposure that requires human attention regardless of whether the system is technically functioning.