Sentry: Building a Distributed Message Broker in Go

Building Sentry: A Distributed Message Broker in Go

Tejas Rastogi is building Sentry, a Kafka-inspired distributed message broker written in Go from scratch, to deeply understand the internal workings of such systems—moving beyond simply using message queues. This project is focused on deterministic behavior and low-level correctness.

Why This Matters

Most engineers rely on abstracted message queue services, lacking insight into their internal mechanisms. This can lead to performance bottlenecks, unexpected failures, and a limited ability to diagnose issues when problems arise. The cost of not understanding these systems can range from minor performance degradation to complete service outages, particularly at scale.

Key Insights

• Custom Binary Wire Protocol: Production brokers avoid JSON over HTTP, opting for framed binary messages for efficiency.
• Append-Only Logs: Append-only logs provide crash safety, sequential disk writes, and deterministic ordering, crucial for reliability.
• Goroutines & Worker Pools: Go’s concurrency model requires careful management with worker pools and backpressure to prevent unbounded resource consumption.

Working Example

type Segment struct {
	baseOffset uint64
	log *os.File
	index *os.File
	timeIndex *os.File
	topic string
	partition uint32
	active bool
}

Practical Applications

• Use Case: Building a custom event streaming platform for a fintech company needing precise control over data durability and latency.
• Pitfall: Assuming managed message queues “just work” without understanding their underlying limitations, leading to scalability issues during peak load.

References:

• https://dev.to/tejas_rastogi_6d73fa2a7a3/sentry-1kna
• https://github.com/tejas2428cse990-svg/Sentry.git