ZooKeeper Learning Project

This repository contains examples demonstrating various patterns and applications of Apache ZooKeeper using Go.

Overview

This project explores several key concepts in distributed systems using ZooKeeper:

1. Kafka Broker Simulation: Demonstrates how Kafka brokers register with ZooKeeper and elect a controller
2. Leader Election: Shows a simple leader election algorithm using ephemeral sequential nodes
3. Consistent Hashing: Implements a distributed hash ring for node discovery and data partitioning

Requirements

• Go 1.21 or higher
• Docker and Docker Compose (for running ZooKeeper)

Getting Started

1. Start the ZooKeeper ensemble:

docker-compose up -d

2. Choose one of the examples to run:

Kafka Broker Simulation

Run multiple broker instances in separate terminals:

# Terminal 1
go run main.go 1

# Terminal 2 
go run main.go 2

# Terminal 3
go run main.go 3

This demonstrates:

• How brokers register under /brokers/ids
• Controller election using the /controller path
• Automatic failover when the controller goes down

Leader Election

Run multiple instances to see leader election in action:

# Terminal 1
go run leader/main.go instance1

# Terminal 2
go run leader/main.go instance2

# Terminal 3
go run leader/main.go instance3

This shows:

• How ephemeral sequential nodes are used for leader election
• Automatic leader election when the current leader fails
• Notification of leader changes

Consistent Hashing

First, start multiple nodes in the hash ring:

# Terminal 1
go run hashing/main.go 1

# Terminal 2
go run hashing/main.go 2

# Terminal 3
go run hashing/main.go 3

Then, in a separate terminal, test the hash ring:

go run hashing/test_client.go

This demonstrates:

• How consistent hashing works with virtual nodes
• How keys are distributed across nodes
• How the system handles node joins and failures

Directory Structure

• /kafka - Kafka broker simulation code
• /leader - Leader election example
• /hashing - Consistent hashing implementation

Key Concepts

• Ephemeral Nodes: Nodes that disappear when a client disconnects
• Sequential Nodes: Nodes with auto-incrementing sequence numbers
• Watches: Notifications for data/children changes
• Consistent Hashing: Even distribution of data across nodes

License

MIT