Skip to content

Commit bb3d4fd

Browse files
decebalclaude
andcommitted
release: v0.10.0 - Leader-Follower Replication, Event-Sourced Metadata & Control Plane v2
## Core: Leader-Follower Replication - WAL-based replication with async/semi-sync/sync modes - WAL shipper (leader) and WAL receiver (follower) over TCP - allsource-sentinel binary for automated failover - Read-only middleware for follower nodes (HTTP 409 on writes) - Internal promote/repoint endpoints for sentinel-driven failover - 11 new Prometheus metrics for replication monitoring - Parquet snapshot catch-up for followers behind WAL range ## Core: Event-Sourced System Metadata - SystemMetadataStore: separate WAL-backed store for _system.* events - SystemBootstrap: 4-stage initialization with first-boot tenant creation - Event-sourced repositories for tenants, audit, and config - Core now dogfoods its own event store for operational metadata - Eliminates PostgreSQL dependency for Core's internal state ## Control Plane: Full Clean Architecture - 31 use cases (tenant CRUD, policy RBAC, operations, schemas, audit, config) - PostgreSQL persistence layer with migrations - Typed Core API client for snapshots, compaction, replay, schemas - JWT auth, async audit logging, policy engine middleware - Routes expanded from ~8 to 40+ with per-route RBAC ## Query Service: Replication-Aware Routing - CoreHealthChecker GenServer polling node health via ETS - Write-to-leader / read-from-followers routing with round-robin - ConsistencyRouting plug (X-Consistency: strong header) - Internal endpoint for sentinel failover notifications ## MCP Server: 36 New Tools - Operational: compact_storage, backup_create/restore, health_deep, wal_status - Multi-tenancy: tenant CRUD, quotas, suspend, export - Schema: register, validate, migrate, infer, diff - Analytics: cohort, correlation, forecast, churn, LTV - Developer: generate_client, mock_events, debug/benchmark_query ## Web Dashboard - Real metrics polling (replaced simulated data) - Live billing API integration - Privacy Policy and Terms of Service pages ## Infrastructure - Rust edition 2024 with workspace dependencies - Release automation (release-plz + git-cliff) - All CI quality gates pass (clippy, fmt, tests, credo, dialyzer) - Version consistency across all 6 services Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent d8cedbe commit bb3d4fd

File tree

259 files changed

+23623
-2489
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

259 files changed

+23623
-2489
lines changed

.github/workflows/ci.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -199,7 +199,7 @@ jobs:
199199
- name: Set up Go
200200
uses: actions/setup-go@v5
201201
with:
202-
go-version: "1.24.13"
202+
go-version: "1.25.6"
203203
cache-dependency-path: apps/control-plane/go.sum
204204

205205
- name: Download dependencies

.github/workflows/release-plz.yml

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
name: Release-plz
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
8+
permissions:
9+
pull-requests: write
10+
contents: write
11+
12+
jobs:
13+
release-plz:
14+
name: Release-plz
15+
runs-on: ubuntu-latest
16+
steps:
17+
- uses: actions/checkout@v4
18+
with:
19+
fetch-depth: 0
20+
21+
- name: Install Rust toolchain
22+
uses: dtolnay/rust-toolchain@nightly
23+
24+
- name: Run release-plz
25+
uses: release-plz/action@v0.5
26+
with:
27+
command: release-pr
28+
env:
29+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
30+
CARGO_REGISTRY_TOKEN: ${{ secrets.CARGO_REGISTRY_TOKEN }}

.github/workflows/release.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -244,7 +244,7 @@ jobs:
244244
- name: Set up Go
245245
uses: actions/setup-go@v5
246246
with:
247-
go-version: "1.24.13"
247+
go-version: "1.25.6"
248248
cache-dependency-path: apps/control-plane/go.sum
249249

250250
- name: Build

.github/workflows/security.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ jobs:
5959
if: matrix.name == 'go'
6060
uses: actions/setup-go@v5
6161
with:
62-
go-version: "1.24.13"
62+
go-version: "1.25.6"
6363
cache-dependency-path: apps/control-plane/go.sum
6464

6565
- name: Go vulnerability check
@@ -110,7 +110,7 @@ jobs:
110110
if: matrix.language == 'go'
111111
uses: actions/setup-go@v5
112112
with:
113-
go-version: "1.24.13"
113+
go-version: "1.25.6"
114114
cache-dependency-path: apps/control-plane/go.sum
115115

116116
- name: Build Go
@@ -192,7 +192,7 @@ jobs:
192192
- name: Set up Go
193193
uses: actions/setup-go@v5
194194
with:
195-
go-version: "1.24.13"
195+
go-version: "1.25.6"
196196
cache-dependency-path: apps/control-plane/go.sum
197197

198198
- name: Install go-licenses

CLAUDE.md

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# CLAUDE.md — AllSource Chronos Monorepo
2+
3+
## Critical: Architecture Facts
4+
5+
**AllSource Core IS the database.** It is a purpose-built Rust event store with full durability:
6+
- **WAL** (Write-Ahead Log): CRC32 checksums, configurable fsync (default 100ms), crash recovery
7+
- **Parquet**: Columnar persistence with Snappy compression, periodic flush
8+
- **DashMap**: In-memory concurrent map for 11.9μs queries and 469K events/sec throughput
9+
10+
**DO NOT** describe Core as "in-memory only", "dumb", "not production-ready", or claim "data is lost on restart." Event data is durable. Only Core's user/tenant metadata (a separate concern) is in-memory — and that responsibility belongs to the Query Service, not Core.
11+
12+
**DO NOT** suggest storing events in PostgreSQL. PostgreSQL is for operational metadata only (users, tenants, API keys, billing). The correct way to improve Core's availability is through Core replication (leader-follower, WAL shipping) — not by adding another database.
13+
14+
## Repository Structure
15+
16+
```
17+
apps/
18+
core/ — Rust event store (AllSource Core)
19+
query-service/ — Elixir/Phoenix API gateway (auth, billing, routing to Core)
20+
web/ — Next.js frontend dashboard
21+
mcp-server-elixir/ — Elixir MCP server (separate from Rust MCP Docker binary)
22+
docs/
23+
proposals/ — Design proposals (e.g., CORE_REPLICATION_DESIGN.md)
24+
current/ — Current architecture docs
25+
```
26+
27+
## Service Architecture
28+
29+
```
30+
Clients → Query Service (Elixir, port 3902) → Core (Rust, port 3900)
31+
| |
32+
PostgreSQL WAL + Parquet + DashMap
33+
(users, tenants, (events, projections,
34+
API keys, billing) snapshots, schemas)
35+
```
36+
37+
- **Core** = the database. Source of truth for all event data.
38+
- **Query Service** = API gateway. Source of truth for users, tenants, billing.
39+
- **PostgreSQL** = operational metadata only. Never for events.
40+
41+
## Core API
42+
43+
All Core endpoints use the `/api/v1/` prefix. Key endpoints:
44+
- `POST /api/v1/events` — ingest event (returns 200, not 201)
45+
- `GET /api/v1/events/query` — query events (returns `{"events": [...], "count": N}`)
46+
- `GET /api/v1/projections` — list projections (returns `{"projections": [...], "total": N}`)
47+
- `GET /api/v1/snapshots` — list snapshots
48+
- `GET /api/v1/schemas` — list schemas
49+
- `GET /health` — health check (note: root path, not /api/v1/health)
50+
- `GET /metrics` — Prometheus metrics
51+
52+
Core wraps responses in maps (`{"events": [...]}`, `{"projections": [...]}`). The Query Service's RustCoreClient unwraps these before passing to controllers.
53+
54+
## Query Service Config
55+
56+
- `CORE_URL` — Core connection URL (not RUST_CORE_URL — clean env var names, no implementation details)
57+
- `CORE_WS_URL` — Core WebSocket URL for real-time streaming
58+
- Config key in Elixir: `:core_url` (not `:rust_core_url`)
59+
60+
## Docker Stack
61+
62+
Defined in the wallet project: `/Users/decebaldobrica/Projects/alphaSigmaPro/wallet/docker/docker-compose.allsource.yml`
63+
64+
Services on `supabase_network_alpha-sigma-pro`:
65+
- `allsource-core-leader` (port 3280 → 3900, replication on 3910)
66+
- `allsource-core-follower-1` (port 3281 → 3900)
67+
- `allsource-core-follower-2` (port 3282 → 3900)
68+
- `allsource-query-service` (port 3283 → 3902)
69+
- `allsource-mcp` (port 3904)
70+
71+
Building Docker on Apple Silicon: native arm64 only — QEMU cross-compilation to linux/amd64 fails on Erlang NIF.
72+
73+
## Scaling Strategy
74+
75+
See `docs/proposals/CORE_REPLICATION_DESIGN.md`:
76+
- Leader-follower replication via WAL shipping
77+
- Query Service routes writes to leader, reads round-robin across followers
78+
- No Raft, no PostgreSQL in the event path, no multi-leader

Cargo.lock

Lines changed: 3 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Cargo.toml

Lines changed: 87 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,88 @@
11
[workspace]
2-
resolver = "2"
3-
members = [
4-
"apps/core",
5-
"tooling/performance",
6-
]
2+
resolver = "3"
3+
members = ["apps/core", "tooling/performance"]
4+
5+
[workspace.dependencies]
6+
7+
# Security & Crypto
8+
aes-gcm = "0.10"
9+
# Internal crates
10+
allsource-core = { path = "apps/core" }
11+
12+
# Error handling
13+
anyhow = "1.0"
14+
argon2 = "0.5"
15+
16+
# Data / Arrow / Parquet
17+
arrow = { version = "53.4", features = ["ipc", "json"] }
18+
arrow-flight = "53.4"
19+
async-trait = "0.1"
20+
21+
# Web framework
22+
axum = { version = "0.8", features = ["json", "ws"] }
23+
axum-extra = { version = "0.10", features = ["typed-header"] }
24+
base64 = "0.22"
25+
bumpalo = { version = "3.16", features = ["collections"] }
26+
bytes = "1.11.1"
27+
28+
# Common utilities
29+
chrono = { version = "0.4", features = ["serde"] }
30+
crc32fast = "1.4"
31+
32+
# Dev / Test
33+
criterion = "0.5"
34+
crossbeam = "0.8"
35+
crossbeam-queue = "0.3"
36+
ctrlc = "3.4"
37+
38+
# Storage & Concurrency
39+
dashmap = "6.1"
40+
datafusion = "44.0"
41+
42+
# Optional features
43+
fastembed = { version = "4" }
44+
45+
# Compression
46+
flate2 = "1.0"
47+
futures = "0.3"
48+
http = "1.0"
49+
instant-distance = { version = "0.6" }
50+
jsonschema = "0.40"
51+
jsonwebtoken = "9.3"
52+
lz4 = "1.28"
53+
parking_lot = "0.12"
54+
parquet = { version = "53.4", features = ["arrow", "async"] }
55+
prometheus = "0.14"
56+
rand = "0.9"
57+
rocksdb = { version = "0.24" }
58+
59+
# Serialization
60+
serde = { version = "1.0", features = ["derive"] }
61+
serde_json = "1.0"
62+
sha2 = "0.10"
63+
simd-json = "0.17"
64+
sqlx = { version = "0.8", features = ["runtime-tokio-rustls", "postgres", "json", "chrono", "uuid"] }
65+
tantivy = { version = "0.22" }
66+
tempfile = "3.23"
67+
testcontainers = "0.23"
68+
testcontainers-modules = { version = "0.11", features = ["postgres"] }
69+
thiserror = "2.0"
70+
time = "0.3.47"
71+
72+
# Async runtime
73+
tokio = { version = "1.48", features = ["full"] }
74+
tokio-stream = "0.1"
75+
toml = "0.8"
76+
tower = "0.5"
77+
tower-http = { version = "0.6", features = ["cors", "trace"] }
78+
79+
# Observability
80+
tracing = "0.1"
81+
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
82+
uuid = { version = "1.19", features = ["v4", "serde"] }
83+
84+
[profile.release]
85+
codegen-units = 1
86+
lto = true
87+
opt-level = 3
88+
strip = true

README.md

Lines changed: 22 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,8 @@ version: "0.9.1"
1414
[![Rust Core](https://img.shields.io/badge/Rust%20Core-v0.9.0-green.svg)](apps/core/)
1515
[![Go Control Plane](https://img.shields.io/badge/Go%20Control%20Plane-v0.9.0-blue.svg)](apps/control-plane/)
1616
[![Elixir Query Service](https://img.shields.io/badge/Elixir%20Query-v0.9.0-purple.svg)](apps/query-service/)
17-
[![MCP Server](https://img.shields.io/badge/MCP%20Server-27%20Tools-orange.svg)](apps/mcp-server-elixir/)
17+
[![MCP Server](https://img.shields.io/badge/MCP%20Server-43%20Tools-orange.svg)](apps/mcp-server-elixir/)
18+
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
1819

1920
High-performance event sourcing platform with distributed architecture and AI-native tooling.
2021

@@ -59,7 +60,7 @@ docker pull ghcr.io/all-source-os/allsource-web:0.9.0
5960

6061
## Project Status
6162

62-
### Current Release: v0.9.0 (February 2026)
63+
### Current Release: v0.9.1 (February 2026)
6364

6465
**Rust Core**
6566
- Event store with 469K events/sec throughput
@@ -83,13 +84,14 @@ docker pull ghcr.io/all-source-os/allsource-web:0.9.0
8384
- WebSocket channels for real-time updates
8485
- Prometheus metrics and APM integration
8586

86-
**MCP Server (27 Tools)**
87+
**MCP Server (43 Tools)**
8788
- AI-native interface via Claude Desktop
88-
- **Event Management Tools**: delete, archive, restore, export, import, clone, merge, split
89-
- **Query Tools**: advanced queries, time series, funnel analysis, anomaly detection
90-
- **Projection Tools**: create, query, list materialized views
89+
- **Core Tools** (19): queries, time series, funnel analysis, anomaly detection, projections, schemas, snapshots
90+
- **Event Management** (8): delete, archive, restore, export, import, clone, merge, split
91+
- **Operational** (10): storage compaction, WAL status, backups, deep health checks, performance reports, audit logs
92+
- **Tenant Management** (6): create, update, usage, quotas, suspend, export
9193
- Dry-run preview mode and audit trails on all operations
92-
- 309 tests passing
94+
- 429 tests passing
9395

9496
**Web Dashboard**
9597
- Modern login/signup with OAuth (Google, GitHub)
@@ -192,22 +194,25 @@ cd apps/mcp-server-elixir && mix test
192194

193195
## Roadmap
194196

195-
### Completed (v0.9.0)
196-
- Event Management Tools (8 new tools)
197-
- Web dashboard UX improvements
197+
### Completed (v0.9.1)
198+
- 43 MCP tools (19 core + 8 event management + 10 operational + 6 tenant)
199+
- Event Management Tools (delete, archive, restore, export, import, clone, merge, split)
200+
- Operational Tools (storage, WAL, backups, health, performance, audit)
201+
- Tenant Management Tools (CRUD, usage, quotas, suspend, export)
202+
- Web dashboard with OAuth, onboarding, billing UI
198203
- Consistent versioning across all services
199204
- OpenAPI specification for Query Service
200205
- WebSocket channels and real-time updates
201206

202207
### In Progress
203-
- SaaS launch preparation
204-
- Multi-region deployment
205-
- Enhanced analytics and reporting
208+
- SaaS launch (self-service signup, billing integration)
209+
- Core WAL-based replication (leader-follower)
210+
- Go Control Plane PostgreSQL migration
206211

207212
### Planned
208-
- GraphQL API layer
209213
- Event sourcing SDK for popular languages
210-
- Self-service onboarding flow
214+
- Multi-region deployment
215+
- GraphQL API layer
211216

212217
**Detailed Roadmaps**:
213218
- [SaaS Launch Roadmap](docs/roadmaps/SAAS_LAUNCH_ROADMAP.md)
@@ -254,5 +259,5 @@ make bump-version # Interactive bump
254259

255260
---
256261

257-
**Last Updated**: February 11, 2026
258-
**Version**: v0.9.0
262+
**Last Updated**: February 12, 2026
263+
**Version**: v0.9.1

apps/control-plane/Dockerfile

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
# No shell, no package manager, no unnecessary utilities
77
# =============================================================================
88

9-
ARG GO_VERSION=1.24.13
9+
ARG GO_VERSION=1.25.6
1010

1111
# =============================================================================
1212
# Stage 1: Build
@@ -73,6 +73,9 @@ WORKDIR /app
7373
# distroless:nonroot runs as UID 65534 by default
7474
COPY --from=builder /app/control-plane /app/control-plane
7575

76+
# Copy migration files for database setup on startup
77+
COPY --from=builder /app/internal/infrastructure/database/migrations /app/migrations
78+
7679
# Expose metrics and API port
7780
EXPOSE 8080
7881

apps/control-plane/auth.go

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -103,8 +103,8 @@ func RoleHasPermission(role entities.Role, perm entities.Permission) bool {
103103
// AuthMiddleware validates JWT tokens and adds auth context to requests
104104
func AuthMiddleware(authClient *AuthClient) gin.HandlerFunc {
105105
return func(c *gin.Context) {
106-
// Skip auth for health endpoints
107-
if c.Request.URL.Path == pathHealth || c.Request.URL.Path == pathMetrics {
106+
// Skip auth for health endpoints and public cluster health
107+
if c.Request.URL.Path == pathHealth || c.Request.URL.Path == pathMetrics || c.Request.URL.Path == "/api/v1/cluster/health" {
108108
c.Next()
109109
return
110110
}
@@ -134,6 +134,8 @@ func AuthMiddleware(authClient *AuthClient) gin.HandlerFunc {
134134

135135
// Store in context
136136
c.Set("auth", authCtx)
137+
c.Set("auth_role", authCtx.Role) // Separate key for cross-package access
138+
c.Set("auth_user_id", authCtx.UserID) // Separate key for cross-package access
137139
c.Next()
138140
}
139141
}

0 commit comments

Comments
 (0)