All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
- Initial release of K2I (Kafka to Iceberg) streaming ingestion engine
- Kafka consumer with smart backpressure and retry logic
- Hot buffer with Apache Arrow for sub-second query freshness
- Iceberg writer with support for REST, Hive, Glue, and Nessie catalogs
- Transaction log for crash recovery and exactly-once semantics
- Automatic maintenance tasks (compaction, snapshot expiration, orphan cleanup)
- Prometheus metrics and health check endpoints
- CLI commands:
ingest,status,maintenance,validate - Comprehensive documentation
- Kafka Consumer: rdkafka-based consumer with cooperative sticky assignment
- Hot Buffer: Arrow RecordBatch with DashMap indexes for O(1) lookups
- Iceberg Writer: Parquet files with Snappy/Zstd/LZ4 compression
- Transaction Log: Append-only log with CRC32 checksums
- Health Checks: Component-level health tracking with Kubernetes-compatible endpoints
- Metrics: Prometheus counters, gauges, and histograms
- Circuit Breaker: Fault tolerance for catalog operations
Initial release.