Distributed Log Aggregation System

A high-performance, distributed log ingestion and search system built in Go, designed to handle 50,000+ log events/sec with batched writes, Kafka-backed durability, and PostgreSQL indexing for fast querying.

• Language: Go
• Messaging: Apache Kafka (bitnami image for local dev)
• Database: PostgreSQL + trigram index for fast text search
• Containerization: Docker Compose
• Design: Event-driven, scalable, and fault-tolerant pipeline

---

Quick links

• Services: ingestion-service/, consumer-service/, search-service/
• Local-compose: docker-compose.yml
• Run script: run.sh

---

Features

• HTTP ingestion endpoint (/ingest) with concurrency-safe batching (flush on 100 logs or 100ms)
• Kafka-backed durability to decouple ingestion from persistence
• Consumer service that writes to PostgreSQL with batched/prepared inserts
• Search API supporting indexed full-text search with optional caching
• Simple deployment with Docker Compose for local testing

---

Architecture

   +----------------------+      +------------------+      +---------------------+
   | Ingestion Service    | ---> | Kafka (logs)     | ---> | Consumer Service    |
   | (HTTP + batching)    |      |                  |      | (Kafka -> Postgres) |
   +----------------------+      +------------------+      +---------------------+
                                                                   |
                                                                   v
                                                           +----------------+
                                                           | PostgreSQL     |
                                                           | (GIN trigram)  |
                                                           +----------------+
                                                                   ^
                                                                   |
                                                           +----------------+
                                                           | Search Service |
                                                           | (HTTP / cache) |
                                                           +----------------+

---

Getting started (local)

Prereqs

• Go 1.18+ (1.22 recommended)
• Docker & Docker Compose
• git

1) Start dependencies

docker compose up -d

2) Initialize DB (one-time)

Connect to Postgres and run:

CREATE TABLE IF NOT EXISTS logs (
  id SERIAL PRIMARY KEY,
  source TEXT,
  message TEXT,
  ts BIGINT,
  created_at TIMESTAMP DEFAULT NOW()
);

CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX IF NOT EXISTS idx_logs_message ON logs USING GIN (message gin_trgm_ops);

3) Run services (manual)

In three terminals:

cd ingestion-service && go run main.go
cd consumer-service && go run main.go
cd search-service && go run main.go

Or use the included run.sh to run everything locally (for demo purposes).

---

run.sh (one-click demo)

run.sh will:

• Bring up Docker Compose
• Run all three services in background using nohup (for demo only)
• Tail logs for quick inspection

Note: run.sh is intended for local demos. For production use, use proper process managers or Docker/Kubernetes.

---

Performance notes & testing

• Batch size: 100 logs / flush interval: 100ms (tuneable via env vars)
• Use ab or custom Go producers to simulate load
• Add pg_trgm index for fast substring search

---

Project structure

.
├── README.md
├── docker-compose.yml
├── run.sh
├── ingestion-service/
├── consumer-service/
└── search-service/

---

Contributing

See CONTRIBUTING.md for guidelines.

---

License

This project is licensed under the MIT License - see the LICENSE file for details.