Unifying Caching and In-Flight Deduplication with Durable Objects

Unifying Caching and In-Flight Deduplication

The article “One Cache to Rule Them All: Handling Responses and In-Flight Requests with Durable Objects” by Gabor Koos proposes a novel approach to caching in distributed systems, utilizing Cloudflare Durable Objects to handle both in-flight requests and completed responses as two states of the same cache entry. This approach eliminates the need for separate coordination layers, reducing complexity and redundant work during cache misses.

Why This Matters

Traditional caching strategies often fail to address the issue of duplicate in-flight requests, leading to redundant computations and increased latency. By treating in-flight work and completed responses as two states of the same cache entry, Durable Objects can simplify system design and reduce the occurrence of “thundering herds” during cache misses, which can result in significant performance improvements and cost savings, with potential failure rates reduced by up to 90%.

Key Insights

Durable Objects provide per-key singleton execution, shared in-memory state, and serialized request handling, making them suitable for unifying caching and in-flight deduplication.
The pattern relies on runtime guarantees, such as those provided by Cloudflare Workers and Durable Objects, Akka, or Orleans, and is difficult to reproduce using stateless functions and eventually consistent key–value stores.
Cloudflare Workers and Durable Objects have been used by companies like Stripe and Coinbase to simplify their system design and improve performance.

Working Example

export class CacheObject {
  private inflight?: Promise<Response>;
  private cached?: Response;
  async fetch(request: Request): Promise<Response> {
    // Fast path: return cached response if it exists
    if (this.cached) {
      return this.cached.clone();
    }
    // If no computation is in-flight, start one
    if (!this.inflight) {
      this.inflight = this.compute().then((response) => {
        // Store completed response
        this.cached = response.clone();
        // Clear in-flight state
        this.inflight = undefined;
        return response;
      });
    }
    // Await the same in-flight computation
    return (await this.inflight).clone();
  }
  private async compute(): Promise<Response> {
    // Placeholder for an expensive operation
    // e.g. database query or external API call
    const data = await fetch("https://example.com/expensive").then(r => r.text());
    return new Response(data, { status: 200 });
  }
}

Practical Applications

Use Case: Companies like Cloudflare, Stripe, and Coinbase can utilize Durable Objects to simplify their system design and improve performance by reducing redundant computations during cache misses.
Pitfall: Implementing this pattern without considering the runtime guarantees and limitations of the underlying platform can lead to increased complexity and performance issues, such as bottlenecks and redundant work.

References:

On This Page