Skip to content

Commit 6a170c9

Browse files
scotwellsscotwells
committed
feat: add high-level architecture document
This commit introduces a new architecture document that can be used to guide the high-level design of the system. This also includes C4 diagrams for the context and container level views of the system.
1 parent e16c831 commit 6a170c9

File tree

8 files changed

+202
-2
lines changed

8 files changed

+202
-2
lines changed

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,5 @@ cluster resources in real-time via audit log events consumed from NATS
66
JetStream, with declarative indexing policies controlling what gets indexed
77
using CEL-based filtering. The service integrates natively with kubectl/RBAC and
88
targets Meilisearch as the search backend.
9+
10+
![](./docs/diagrams/SearchServiceContext.png)

‎Taskfile.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,10 @@
11
version: '3'
22

3+
includes:
4+
docs:
5+
taskfile: ./docs/Taskfile.yaml
6+
dir: ./docs
7+
38
vars:
49
TOOL_DIR: "{{.USER_WORKING_DIR}}/bin"
510
IMAGE_NAME: "ghcr.io/datum-cloud/search"
@@ -116,3 +121,10 @@ tasks:
116121
- go vet ./...
117122
- echo "✅ Vet complete"
118123
silent: true
124+
125+
# Architecture diagram tasks
126+
diagrams:
127+
desc: Generate architecture diagrams from PlantUML
128+
cmds:
129+
- task: docs:diagrams
130+
silent: true

‎docs/Taskfile.yaml‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
version: '3'
2+
3+
vars:
4+
DIAGRAMS_DIR: "{{.USER_WORKING_DIR}}/docs/diagrams"
5+
OUTPUT_FORMAT: "png"
6+
7+
tasks:
8+
generate:
9+
desc: Generate all documentation artifacts (diagrams, etc.)
10+
cmds:
11+
- task: diagrams:render
12+
silent: true
13+
14+
diagrams:
15+
desc: Generate all architecture diagrams from PlantUML
16+
cmds:
17+
- task: diagrams:render
18+
silent: true
19+
20+
diagrams:render:
21+
desc: Render PlantUML diagrams to PNG format using Docker
22+
cmds:
23+
- |
24+
set -e
25+
echo "Rendering PlantUML diagrams..."
26+
echo ""
27+
28+
# Check if PlantUML files exist
29+
if [ ! -f "{{.DIAGRAMS_DIR}}/context.puml" ] || [ ! -f "{{.DIAGRAMS_DIR}}/container.puml" ]; then
30+
echo "❌ Error: PlantUML source files not found in {{.DIAGRAMS_DIR}}"
31+
exit 1
32+
fi
33+
34+
# Render using Docker (no local installation required)
35+
docker run --rm \
36+
-v "{{.DIAGRAMS_DIR}}":/data \
37+
plantuml/plantuml:latest \
38+
-t{{.OUTPUT_FORMAT}} \
39+
/data/*.puml
40+
41+
echo ""
42+
echo "✅ Diagrams rendered in {{.DIAGRAMS_DIR}}"
43+
echo ""
44+
echo "Generated files:"
45+
ls -1 {{.DIAGRAMS_DIR}}/*.{{.OUTPUT_FORMAT}} 2>/dev/null | xargs -n1 basename || echo "No output files found"
46+
silent: true
47+
48+
diagrams:clean:
49+
desc: Remove generated diagram files
50+
cmds:
51+
- |
52+
rm -f {{.DIAGRAMS_DIR}}/*.png {{.DIAGRAMS_DIR}}/*.svg
53+
echo "✅ Generated diagram files removed"
54+
silent: true
55+
56+
diagrams:validate:
57+
desc: Validate PlantUML syntax using Docker
58+
cmds:
59+
- |
60+
set -e
61+
echo "Validating PlantUML diagrams..."
62+
docker run --rm \
63+
-v "{{.DIAGRAMS_DIR}}":/data \
64+
plantuml/plantuml:latest \
65+
-syntax \
66+
/data/*.puml
67+
echo "✅ All diagrams are valid"
68+
silent: true

‎docs/architecture.md‎

Lines changed: 74 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,75 @@
1-
# Architecture
1+
# Search Architecture
22

3-
TODO: Document service architecture
3+
## Overview
4+
5+
The Search service is a Kubernetes-native API built on the aggregated
6+
API server framework that provides advanced resource discovery capabilities
7+
through field filtering and full-text search. It enables platform users to
8+
efficiently query and locate resources across the cluster using powerful
9+
indexing and real-time event processing.
10+
11+
## Architecture Diagram
12+
13+
> [!NOTE]
14+
>
15+
> Below is a [C4 container diagram][c4] of the service and it's dependencies.
16+
> This is meant to model individual components in the system and their
17+
> responsibilities. It does not aim to provide visibility into external system
18+
> components that may be a dependency of this system.
19+
20+
<p align="center">
21+
<img src="./diagrams/SearchServiceContainers.png" alt="Search service component software architecture diagram">
22+
</p>
23+
24+
[c4]: https://c4model.com
25+
26+
## Components
27+
28+
### Search API Server
29+
30+
**Purpose**: Expose search capabilities as native Kubernetes APIs
31+
32+
**Responsibilities**:
33+
- Register custom API endpoints under `search.miloapis.com/v1alpha1`
34+
- Handle authentication and authorization via Kubernetes RBAC
35+
- Provide RESTful API for search queries
36+
- Manage custom resource definitions for the search service
37+
38+
**Query Types**:
39+
- **Field Filtering**: Exact match, prefix, range queries on structured fields
40+
- **Full-Text Search**: Fuzzy matching, phrase queries, relevance scoring
41+
42+
### Resource Indexer
43+
44+
**Purpose**: Real-time indexing of platform resources from audit logs
45+
46+
**Responsibilities**:
47+
- Subscribe to NATS JetStream audit log topic
48+
- Filter events based on active index policies
49+
- Evaluate CEL expressions for conditional indexing
50+
- Extract and transform resource data for indexing
51+
- Write to index backend with proper error handling and retries
52+
- Manage index lifecycle (creation, updates, deletion)
53+
- Bootstrap indexes from existing state
54+
55+
### Controller Manager
56+
57+
**Purpose**: Manages and validates resources for the search service
58+
59+
**Responsibilities**:
60+
- Validates and activates index policies
61+
62+
### Index Backend Storage
63+
64+
**Purpose**: High-performance full-text search and indexing
65+
66+
**Responsibilities**:
67+
- Structured metadata (namespace, name, labels, annotations) filtering
68+
- Full-text searchable content
69+
70+
> [!NOTE]
71+
>
72+
> We're targeting [Meilisearch] as our first integration backend for indexed
73+
> storage.
74+
75+
[Meilisearch]: https://www.meilisearch.com

‎docs/diagrams/SearchServiceContainers.png‎

43.4 KB
Loading

‎docs/diagrams/SearchServiceContext.png‎

26.9 KB
Loading

‎docs/diagrams/container.puml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
@startuml SearchServiceContainers
2+
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
3+
4+
Person(platformUser, "Platform User", "An operator who needs to query and locate platform resources")
5+
6+
System_Ext(platformAPI, "Platform API", "Manages cluster resources and provides aggregation layer for custom APIs")
7+
System_Ext(eventBus, "Event Streaming", "Event stream of audit logs collected from the apiserver")
8+
9+
System_Boundary(searchService, "Search Service") {
10+
Container(searchAPI, "Search APIServer", "Go, Kubernetes API Server Framework", "Provides a platform native API for searching resources")
11+
Container(indexerService, "Resource Indexer", "Go", "Indexes platform resources based on index policies")
12+
Container(controllerManager, "Controller Manager", "Go, Kubernetes Controller", "Manages index policies in the search service")
13+
System(indexBackend, "Index Backend Storage", "High-performance full-text search engine for indexed resources")
14+
}
15+
16+
Rel_D(platformUser, searchAPI, "Submits search queries and manages index policies", "HTTPS/JSON/UI")
17+
Rel(searchAPI, indexBackend, "Executes search queries", "HTTPS/JSON")
18+
Rel_L(controllerManager, searchAPI, "Watches search resources", "HTTPS/Watch API")
19+
20+
Rel_L(indexerService, eventBus, "Consumes events from", "NATS Protocol")
21+
Rel_U(indexerService, searchAPI, "Retrieves index policies from", "HTTPS")
22+
Rel_D(platformAPI, eventBus, "Sends audit log events to", "HTTPS")
23+
Rel_R(indexerService, platformAPI, "Bootstraps indexes from resources in", "HTTPS")
24+
Rel_R(indexerService, indexBackend, "Writes indexed documents", "HTTPS/JSON")
25+
26+
@enduml

‎docs/diagrams/context.puml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
@startuml SearchServiceContext
2+
!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Context.puml
3+
4+
left to right direction
5+
6+
Person(platformUser, "Platform User", "An operator who needs to query and locate platform resources")
7+
8+
System(searchService, "Search Service", "Provides advanced resource discovery through field filtering and full-text search of platform resources")
9+
10+
System_Ext(platformAPI, "Platform API", "Manages core resources, authn, authz, and provides aggregation layer for custom APIs")
11+
System_Ext(eventStream, "Event Streaming System", "Streams audit log events from the platform API")
12+
13+
Rel_R(platformUser, searchService, "Searches resources using CLI, UI, or API clients", "HTTPS/JSON")
14+
Rel(platformUser, platformAPI, "Manages platform resources", "kubectl/HTTPS")
15+
16+
Rel_D(searchService, platformAPI, "Indexes resources from", "HTTPS")
17+
Rel(searchService, eventStream, "Subscribes to audit log events", "NATS Protocol")
18+
Rel(platformAPI, eventStream, "Publishes audit logs to")
19+
20+
@enduml

0 commit comments

Comments
 (0)