Skip to content

Commit fef5c57

Browse files
Gezi-lzqGezi-lzq
authored
feat(dev): add local development environment (#3218)
* feat(devbox): add zero-config local development environment Replace local-playground with devbox — a lightweight, layered local dev environment for AutoMQ developers. Features: - just start / just start-build: single node or cluster (up to 5 nodes) - Docker Compose profiles: single, cluster, tabletopic, analytics - 4-layer config system: defaults → features → custom → auto-generated - just bin: direct passthrough to any bin/ command in container - Shortcuts: topic-list, topic-create, produce, consume - Chaos engineering: chaos-delay, chaos-loss, chaos-reset - JDWP remote debugging on every node - Stable auto-generated CLUSTER_ID Dependencies: just + docker + bash * fix(devbox): correct spacing in topic description output * feat(devbox): update CI to test 4-node cluster configuration * fix(devbox): update CI to depend on lint instead of smoke for full job * feat(devbox): add concurrency settings to CI workflow * fix(devbox): remove interactive flag from docker exec command * feat(devbox): update advertised listeners for controller nodes in configuration * docs(devbox): update README to clarify role and node identity configuration * refactor(devbox): improve security, CI coverage, and documentation - Remove privileged flag, use NET_ADMIN capability only - Add Docker image build to build command - Clean up apt lists in Dockerfile to reduce image size - Remove unused SKIP variable from bin command - Add tabletopic and analytics profiles to CI validation - Remove duplicate 4-node cluster test - Fix PartitionCount grep pattern (add space) - Update documentation to clarify build includes image and code - Add note about container-internal access design - Add concurrency control to cancel old CI runs * feat(devbox): refactor justfile to use KAFKA_OPTS for Kafka commands * feat(devbox): add JMX operations and update configurations for Kafka * feat(devbox): update KAFKA_OPTS to include JMX_PORT for improved monitoring * feat(devbox): set JMX_PORT for Kafka containers and enhance restart command * feat(devbox): set JMX_PORT in docker-compose for Kafka services * feat(devbox): add build-image command to rebuild Docker images with specified nodes
1 parent ceb6943 commit fef5c57

File tree

11 files changed

+913
-0
lines changed

11 files changed

+913
-0
lines changed

‎.github/workflows/devbox-ci.yml‎

Lines changed: 219 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,219 @@
1+
name: DevBox CI
2+
3+
on:
4+
pull_request:
5+
paths:
6+
- 'devbox/**'
7+
workflow_dispatch:
8+
inputs:
9+
level:
10+
description: 'Test level'
11+
type: choice
12+
options: [lint, smoke, full]
13+
default: smoke
14+
15+
concurrency:
16+
group: ${{ github.workflow }}-${{ github.ref }}
17+
cancel-in-progress: true
18+
19+
permissions:
20+
contents: read
21+
22+
env:
23+
AUTOMQ_DEV_HOME: ${{ github.workspace }}
24+
25+
jobs:
26+
lint:
27+
name: Lint
28+
runs-on: ubuntu-latest
29+
steps:
30+
- uses: actions/checkout@v4
31+
32+
- name: Install just
33+
uses: extractions/setup-just@v2
34+
35+
- name: Validate justfile
36+
working-directory: devbox
37+
run: just --list
38+
39+
- name: Validate docker-compose
40+
working-directory: devbox
41+
run: |
42+
for p in single cluster cluster4 cluster5 tabletopic analytics; do
43+
echo "=== Profile: $p ==="
44+
docker compose -f docker-compose.yml --profile $p config --services
45+
done
46+
47+
- name: Validate config generation
48+
working-directory: devbox
49+
run: |
50+
# Generate cluster-id without kafka-storage.sh
51+
mkdir -p .devbox
52+
echo "test-cluster-id" > .devbox/cluster-id
53+
54+
# Single node, no features
55+
just generate-config 1 ""
56+
grep -q "node.id=0" .devbox/config/node-0.properties
57+
grep -q "controller.quorum.voters=0@node-0:9093" .devbox/config/node-0.properties
58+
! grep -q "broker.rack" .devbox/config/node-0.properties
59+
echo "✓ Single node config OK"
60+
61+
# 3-node cluster with zerozone
62+
just generate-config 3 "zerozone"
63+
grep -q "broker.rack=az-0" .devbox/config/node-0.properties
64+
grep -q "broker.rack=az-1" .devbox/config/node-1.properties
65+
grep -q "broker.rack=az-2" .devbox/config/node-2.properties
66+
grep -q "zonerouter" .devbox/config/node-0.properties
67+
grep -q "controller.quorum.voters=0@node-0:9093,1@node-1:9093,2@node-2:9093" .devbox/config/node-0.properties
68+
echo "✓ Cluster + zerozone config OK"
69+
70+
# Tabletopic feature
71+
just generate-config 1 "tabletopic"
72+
grep -q "automq.table.topic.catalog.type=rest" .devbox/config/node-0.properties
73+
echo "✓ Tabletopic config OK"
74+
75+
# Combined features
76+
just generate-config 1 "tabletopic zerozone"
77+
grep -q "automq.table.topic" .devbox/config/node-0.properties
78+
grep -q "zonerouter" .devbox/config/node-0.properties
79+
grep -q "broker.rack" .devbox/config/node-0.properties
80+
echo "✓ Combined features config OK"
81+
82+
smoke:
83+
name: Smoke
84+
needs: lint
85+
runs-on: ubuntu-latest
86+
timeout-minutes: 20
87+
steps:
88+
- uses: actions/checkout@v4
89+
90+
- uses: actions/setup-java@v4
91+
with:
92+
distribution: temurin
93+
java-version: 17
94+
95+
- uses: extractions/setup-just@v2
96+
97+
- name: Gradle build
98+
run: ./gradlew :core:build :tools:build :shell:build -x test
99+
100+
- name: Start single node
101+
working-directory: devbox
102+
run: just start
103+
104+
- name: Wait for healthy
105+
run: |
106+
for i in $(seq 1 30); do
107+
S=$(docker inspect --format='{{.State.Health.Status}}' node-0 2>/dev/null)
108+
[ "$S" = "healthy" ] && echo "✓ node-0 healthy" && exit 0
109+
sleep 5
110+
done
111+
echo "Timeout waiting for node-0" && docker logs node-0 | tail -50 && exit 1
112+
113+
- name: Topic operations
114+
working-directory: devbox
115+
run: |
116+
just topic-create ci-test
117+
just topic-list 2>&1 | grep ci-test
118+
just topic-describe ci-test 2>&1 | grep "PartitionCount: 1"
119+
echo "✓ Topic operations OK"
120+
121+
- name: Produce and consume
122+
run: |
123+
echo -e "ci-msg-1\nci-msg-2" | docker exec -i node-0 bash -c \
124+
"KAFKA_HEAP_OPTS='-Xms256m -Xmx256m' KAFKA_JVM_PERFORMANCE_OPTS='' /opt/automq/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic ci-test"
125+
docker exec node-0 bash -c \
126+
"KAFKA_HEAP_OPTS='-Xms256m -Xmx256m' KAFKA_JVM_PERFORMANCE_OPTS='' /opt/automq/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic ci-test --from-beginning --max-messages 2 --timeout-ms 15000" \
127+
2>&1 | grep ci-msg-1
128+
echo "✓ Produce/consume OK"
129+
130+
- name: Bin passthrough
131+
working-directory: devbox
132+
run: |
133+
just bin kafka-broker-api-versions.sh --bootstrap-server localhost:9092 2>&1 | grep -q "ApiVersion"
134+
echo "✓ Bin passthrough OK"
135+
136+
- name: JMX operations
137+
working-directory: devbox
138+
run: |
139+
just jmx -e domains 2>&1 | grep -q "kafka.server"
140+
just jmx -e 'beans -d kafka.server' 2>&1 | grep -q "kafka.server"
141+
echo "✓ JMX operations OK"
142+
143+
- name: Stop
144+
if: always()
145+
working-directory: devbox
146+
run: just stop
147+
148+
full:
149+
name: Full
150+
needs: lint
151+
runs-on: ubuntu-latest
152+
timeout-minutes: 30
153+
steps:
154+
- uses: actions/checkout@v4
155+
156+
- uses: actions/setup-java@v4
157+
with:
158+
distribution: temurin
159+
java-version: 17
160+
161+
- uses: extractions/setup-just@v2
162+
163+
- name: Gradle build
164+
run: ./gradlew :core:build :tools:build :shell:build -x test
165+
166+
- name: Test tabletopic
167+
working-directory: devbox
168+
run: |
169+
just start 1 tabletopic
170+
for i in $(seq 1 30); do
171+
S=$(docker inspect --format='{{.State.Health.Status}}' node-0 2>/dev/null)
172+
[ "$S" = "healthy" ] && break; sleep 5
173+
done
174+
curl -sf http://localhost:8181/v1/config > /dev/null
175+
echo "✓ Iceberg REST OK"
176+
for i in $(seq 1 30); do
177+
curl -sf http://localhost:8081/ > /dev/null && echo "✓ Schema Registry OK" && break
178+
sleep 5
179+
done
180+
grep -q "automq.table.topic" .devbox/config/node-0.properties
181+
echo "✓ Tabletopic config merged"
182+
just stop
183+
184+
- name: Test 4-node cluster (3 controller + 1 broker)
185+
working-directory: devbox
186+
run: |
187+
just start 4
188+
# Verify role configs
189+
grep -q "process.roles=broker,controller" .devbox/config/node-0.properties
190+
grep -q "process.roles=broker,controller" .devbox/config/node-1.properties
191+
grep -q "process.roles=broker,controller" .devbox/config/node-2.properties
192+
grep -q "^process.roles=broker$" .devbox/config/node-3.properties
193+
grep -q "controller.quorum.voters=0@node-0:9093,1@node-1:9093,2@node-2:9093" .devbox/config/node-0.properties
194+
echo "✓ Role configs correct"
195+
# Wait for healthy
196+
for node in 0 1 2 3; do
197+
for i in $(seq 1 30); do
198+
S=$(docker inspect --format='{{.State.Health.Status}}' node-${node} 2>/dev/null)
199+
[ "$S" = "healthy" ] && echo "✓ node-${node} healthy" && break
200+
sleep 5
201+
done
202+
done
203+
# Test topic operations
204+
just topic-create mixed-ci-test
205+
just topic-describe mixed-ci-test 2>&1 | grep -E "Leader: [0123]"
206+
echo "✓ Mixed cluster (3 controller + 1 broker) OK"
207+
just stop
208+
209+
- name: Test zerozone config
210+
working-directory: devbox
211+
run: |
212+
just start 5 zerozone
213+
grep -q "broker.rack=az-0" .devbox/config/node-0.properties
214+
grep -q "broker.rack=az-1" .devbox/config/node-1.properties
215+
grep -q "broker.rack=az-2" .devbox/config/node-2.properties
216+
grep -q "broker.rack=az-0" .devbox/config/node-3.properties
217+
grep -q "broker.rack=az-1" .devbox/config/node-4.properties
218+
echo "✓ ZeroZone broker.rack round-robin OK"
219+
just stop

‎devbox/.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
.devbox/
2+
config/custom.properties
3+
__pycache__/

‎devbox/Dockerfile‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
ARG jdk_version=eclipse-temurin:17-jdk-jammy
2+
FROM $jdk_version
3+
4+
ENV DEBIAN_FRONTEND=noninteractive
5+
6+
RUN apt-get update && apt-get install -y \
7+
curl jq iproute2 iputils-ping net-tools wget \
8+
&& wget -O /usr/local/bin/jmxterm.jar https://github.com/jiaqi/jmxterm/releases/download/v1.0.4/jmxterm-1.0.4-uber.jar \
9+
&& apt-get -y clean && rm -rf /var/lib/apt/lists/*
10+
11+
WORKDIR /opt/automq
12+
CMD tail -f /dev/null

‎devbox/README.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
# DevBox - AutoMQ Local Development Environment
2+
3+
Zero-configuration local dev environment for AutoMQ. One command to start a Kafka cluster with S3 storage (MinIO), JDWP debugging, and optional features.
4+
5+
## Prerequisites
6+
7+
- Docker & Docker Compose
8+
- [just](https://github.com/casey/just) command runner
9+
- JDK 17+
10+
11+
## Quick Start
12+
13+
```bash
14+
cd devbox
15+
just start-build # Build image & code + single node
16+
just start-build 3 # Build image & code + 3-node cluster
17+
```
18+
19+
After first build, iterate faster with:
20+
```bash
21+
just start # Single node (skip build)
22+
just start 3 # 3-node cluster (skip build)
23+
```
24+
25+
DevBox automatically generates a stable CLUSTER_ID, node configs, and starts MinIO + AutoMQ in Docker with JDWP enabled.
26+
27+
## Commands
28+
29+
### Lifecycle
30+
31+
| Command | Description |
32+
|---------|-------------|
33+
| `just start` | Single node (skip build) |
34+
| `just start 3` | 3-node cluster |
35+
| `just start 1 telemetry` | Single node + OpenTelemetry metrics |
36+
| `just start 5 tabletopic` | 5-node + TableTopic |
37+
| `just start-build` | Build image & code, then start single node |
38+
| `just start-build 3 tabletopic analytics` | Build + full stack |
39+
| `just stop` | Stop all services |
40+
| `just restart` | Rebuild and restart |
41+
| `just status` | Show service status |
42+
| `just logs` | Tail all logs |
43+
| `just logs 0` | Tail node 0 logs |
44+
45+
### Bin (direct passthrough)
46+
47+
Run any `bin/` command inside a node container:
48+
49+
```bash
50+
just bin kafka-topics.sh --bootstrap-server localhost:9092 --list
51+
just bin kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type brokers
52+
just bin --node 1 kafka-broker-api-versions.sh --bootstrap-server localhost:9092
53+
```
54+
55+
> **Note**: The cluster is designed for container-internal access. To interact with Kafka from your host machine, use `just bin` commands or the shortcuts below (`topic-*`, `produce`, `consume`).
56+
57+
### Shortcuts
58+
59+
| Command | Description |
60+
|---------|-------------|
61+
| `just topic-list` | List topics |
62+
| `just topic-create my-topic --partitions 16` | Create topic with 16 partitions |
63+
| `just topic-describe my-topic` | Describe topic |
64+
| `just produce my-topic` | Interactive producer |
65+
| `just consume my-topic --from-beginning` | Consumer |
66+
67+
### Chaos
68+
69+
| Command | Description |
70+
|---------|-------------|
71+
| `just chaos-delay` | 200ms delay on node-0 |
72+
| `just chaos-delay 500 node-1` | 500ms delay on node-1 |
73+
| `just chaos-loss` | 5% packet loss on node-0 |
74+
| `just chaos-loss 10 node-1` | 10% packet loss on node-1 |
75+
| `just chaos-reset` | Remove all rules |
76+
| `just chaos-status` | Show rules |
77+
78+
## Port Allocation
79+
80+
| Node | Kafka | Controller | JDWP |
81+
|------|-------|------------|------|
82+
| 0 | 9092 | 9093 | 5005 |
83+
| 1 | 19092 | 19093 | 5006 |
84+
| 2 | 29092 | 29093 | 5007 |
85+
| 3 | 39092 | 39093 | 5008 |
86+
| 4 | 49092 | 49093 | 5009 |
87+
88+
Other services:
89+
- MinIO Console: http://localhost:9001 (admin/password)
90+
- Schema Registry: http://localhost:8081 (with tabletopic)
91+
- Iceberg REST: http://localhost:8181 (with tabletopic)
92+
- Spark: http://localhost:8888 (with analytics)
93+
- Trino: http://localhost:8090 (with analytics)
94+
95+
## IDEA Remote Debugging
96+
97+
1. Run → Edit Configurations → + → Remote JVM Debug
98+
2. Host: `localhost`, Port: `5005` (node 0)
99+
3. Start DevBox, then attach debugger
100+
101+
## Configuration
102+
103+
5-layer config system (low → high priority):
104+
105+
1. `config/defaults.properties` — MinIO, KRaft basics
106+
2. Role — process.roles, listeners, advertised.listeners (auto: controller for node 0-2, broker-only for 3+)
107+
3. Node identity — node.id, cluster.id, quorum.voters (auto)
108+
4. `config/features/*.properties` — tabletopic, zerozone, telemetry (applied via start args)
109+
5. `config/custom.properties` — your overrides, highest priority (gitignored)
110+
111+
To customize, create `config/custom.properties`:
112+
113+
```properties
114+
num.partitions=4
115+
log.retention.hours=24
116+
```
117+

‎devbox/config/defaults.properties‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# AutoMQ DevBox - Default Configuration
2+
# Layer 1: Base settings for local development with MinIO
3+
4+
# KRaft mode
5+
controller.listener.names=CONTROLLER
6+
inter.broker.listener.name=PLAINTEXT
7+
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
8+
9+
# MinIO S3 storage
10+
elasticstream.enable=true
11+
s3.data.buckets=0@s3://automq-data?region=us-east-1&endpoint=http://minio:9000&pathStyle=true
12+
s3.ops.buckets=1@s3://automq-ops?region=us-east-1&endpoint=http://minio:9000&pathStyle=true
13+
s3.wal.path=0@s3://automq-data?region=us-east-1&endpoint=http://minio:9000&pathStyle=true
14+
15+
# Internal topics (single-node defaults)
16+
offsets.topic.replication.factor=1
17+
transaction.state.log.replication.factor=1
18+
transaction.state.log.min.isr=1
19+
20+
# Log
21+
log.dirs=/tmp/kraft-combined-logs
22+
num.partitions=1
23+
24+
# Auto Balancer
25+
metric.reporters=kafka.autobalancer.metricsreporter.AutoBalancerMetricsReporter

‎devbox/config/features/tabletopic.properties‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# AutoMQ DevBox - TableTopic Feature
2+
# Layer 2: Enables Iceberg table topic support
3+
4+
automq.table.topic.catalog.type=rest
5+
automq.table.topic.catalog.uri=http://rest:8181
6+
automq.table.topic.catalog.warehouse=s3://warehouse/wh/
7+
automq.table.topic.namespace=default
8+
automq.table.topic.schema.registry.url=http://schema-registry:8081

‎devbox/config/features/telemetry.properties‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# AutoMQ DevBox - Telemetry Feature
2+
# Enables OpenTelemetry metrics export
3+
# Change endpoint to your OTel Collector address before use
4+
5+
s3.telemetry.metrics.exporter.uri=otlp://?endpoint=http://localhost:4318&protocol=http
6+
s3.telemetry.metrics.level=INFO
7+
s3.telemetry.exporter.report.interval.ms=30000

0 commit comments

Comments
 (0)