Merge pull request #2 from numtide/multi-go-module-setup

Multi go module setup

Author: rytswd

.gitignore

@@ -7,3 +7,6 @@ result-*
 
 # generic
 dist/
+
+# Go
+go.work

config/.gitkeep

@@ -31,9 +31,14 @@ The operator creates and manages all necessary Kubernetes resources for these co
 
 ```
 multigres-operator/
-├── api/v1alpha1/           # CRD definitions
+├── go.mod                  # Root module
 ├── cmd/multigres-operator/ # Main entry point
-└── internal/               # Controller and resource builders
+└── pkg/                    # Multi-module structure
+    ├── cluster-handler/    # Cluster orchestration (separate module)
+    ├── data-handler/       # Data plane management (separate module)
+    └── resource-handler/   # Component resources (separate module)
+
+Note: go.work can be created locally for development. It is in .gitignore
 ```
 
 ### Documentation and Configuration

docs/README.md

@@ -40,36 +40,68 @@
 
 ```
 multigres-operator/
-├── api/v1alpha1/           # CRD definitions and types
+├── go.mod                  # Root module (primarily for cmd)
 ├── cmd/multigres-operator/ # Main entry point
-├── internal/
-│   ├── controller/         # Main reconciler and component reconcilers
-│   │   ├── etcd/
-│   │   ├── multigateway/
-│   │   ├── multiorch/
-│   │   └── multipooler/
-│   ├── resources/          # Pure resource builder functions
-│   ├── webhook/            # Admission webhooks (defaulting, validation)
-│   └── testutil/           # Test helpers and utilities
+├── pkg/
+│   ├── cluster-handler/    # Cluster-level orchestration (separate Go module)
+│   │   ├── go.mod
+│   │   └── controller/
+│   │       └── multigrescluster/  # MultigresCluster reconciler
+│   ├── data-handler/       # Data plane management (separate Go module)
+│   │   ├── go.mod
+│   │   └── controller/
+│   │       └── cell/       # Cell reconciler
+│   └── resource-handler/   # Component resources (separate Go module)
+│       ├── go.mod
+│       └── controller/
+│           ├── multigateway/
+│           ├── multiorch/
+│           ├── multipooler/
+│           └── etcd/
 ├── config/                 # Kubernetes manifests for operator deployment
 ├── docs/                   # Architecture, conventions, and development guides
 └── plans/                  # Planning documents
-```
-
-### API Layer (`api/v1alpha1`)
-- **Purpose**: Defines Multigres custom resource schema
-- **Components**: MultigresSpec, MultigresStatus, component specs (MultiGatewaySpec, MultiOrchSpec, MultiPoolerSpec, EtcdSpec)
-- **Validation**: Kubebuilder markers for OpenAPI validation and defaults
 
-### Controller Layer (`internal/controller`)
-- **MultigresReconciler**: Main controller - manages lifecycle, finalizers, status aggregation
-- **Component Reconcilers**: etcd, multigateway, multiorch, multipooler reconcilers run in parallel
-- **Responsibilities**: Create/update resources, check component health, update status
+Note: go.work file can be created locally for development convenience. It is in .gitignore.
+```
 
-### Resource Layer (`internal/resources`)
-- **Pure Functions**: Build Kubernetes manifests (Deployments, StatefulSets, Services, HPAs)
-- **Label Management**: Consistent label generation for resource selection
-- **No Side Effects**: Same input always produces same output
+### Multi-Module Architecture
+
+The operator is organized into three independent Go modules with clear separation of concerns. Developers can use **Go workspaces** locally (`go.work`) for easier cross-module development, though this file is not committed to the repository:
+
+#### Cluster Handler Module (`pkg/cluster-handler`)
+- **Purpose**: High-level cluster orchestration
+- **Responsibilities**:
+  - Uses the `MultigresCluster` custom resource as its primary input
+  - Coordinates creation of child resources (MultiGateway, MultiOrch, MultiPooler, Etcd, Cell)
+  - Aggregates status from child resources
+  - Handles cluster-wide lifecycle (finalizers, upgrades)
+- **Independence**: Can be tested and versioned separately from resource handlers
+
+#### Data Handler Module (`pkg/data-handler`)
+- **Purpose**: Data plane management and topology
+- **Responsibilities**:
+  - Uses the `Cell` custom resource as its primary input (data topology abstraction)
+  - Handles data plane configuration and routing
+  - Coordinates with Vitess/Multigres data components
+- **Independence**: Data plane logic isolated from control plane and resource management
+
+#### Resource Handler Module (`pkg/resource-handler`)
+- **Purpose**: Individual component resource management
+- **Responsibilities**:
+  - Uses the `MultiGateway`, `MultiOrch`, `MultiPooler`, `Etcd` custom resources as its primary inputs
+  - Creates and reconciles Kubernetes resources (Deployments, StatefulSets, Services, HPAs)
+  - Implements pure resource builder functions
+  - Component-specific health checking and status reporting
+- **Independence**: Component reconcilers can evolve independently of cluster orchestration
+
+### Benefits of Multi-Module Structure
+
+- **Clear Boundaries**: Each module has distinct responsibilities and can be understood independently
+- **Independent Testing**: Unit and integration tests scoped to each module
+- **Parallel Development**: Teams can work on different modules without conflicts
+- **Selective Imports**: Main binary only imports what it needs from each module
+- **Version Flexibility**: Modules can evolve at different paces (though currently developed in lockstep)
 
 ## Core Components
 
@@ -85,14 +117,14 @@ The operator follows a standard Kubernetes reconciliation pattern:
 
 ### Component Reconcilers
 
-Each Multigres component has its own reconciler:
+The **resource-handler** module contains individual reconcilers for each Multigres component:
 
-- **etcd Reconciler**: Manages StatefulSet for etcd cluster, headless and client Services
-- **multigateway Reconciler**: Manages Deployment, Service, and optional HPA for MultiGateway
-- **multiorch Reconciler**: Manages Deployment and optional HPA for MultiOrch
-- **multipooler Reconciler**: Manages StatefulSet with multi-container pods (pooler, pgctld, postgres), and optional HPA
+- **etcd Reconciler** (`pkg/resource-handler/controller/etcd`): Manages StatefulSet for etcd cluster, headless and client Services
+- **multigateway Reconciler** (`pkg/resource-handler/controller/multigateway`): Manages Deployment, Service, and optional HPA for MultiGateway
+- **multiorch Reconciler** (`pkg/resource-handler/controller/multiorch`): Manages Deployment and optional HPA for MultiOrch
+- **multipooler Reconciler** (`pkg/resource-handler/controller/multipooler`): Manages StatefulSet with multi-container pods (pooler, pgctld, postgres), and optional HPA
 
-All component reconcilers run in parallel.
+All component reconcilers run in parallel and are independent of each other.
 
 ### Resource Builders
 
@@ -146,8 +178,8 @@ Current preference is init container pattern (same as Istio, NGINX Ingress), but
 ## Technology Stack
 
 ### Language and Runtime
-- **Language**: Go 1.24+
-- **Key Features**: Concurrency, interfaces, strong typing
+- **Language**: Go 1.25+
+- **Key Features**: Concurrency, interfaces, strong typing, workspaces for multi-module projects
 
 ### Key Dependencies
 - **Framework**: Kubebuilder v3 - scaffolding and patterns for Kubernetes operators

docs/architecture.md

@@ -5,13 +5,13 @@
 ## Development Environment
 
 ### Language Configuration
-- **Language**: Go 1.24+
-- **Build System**: Make for task orchestration, Go modules for dependencies
-- **Project Structure**: Standard Kubebuilder layout - `api/`, `cmd/`, `internal/`, `config/`
+- **Language**: Go 1.25+
+- **Build System**: Make for task orchestration, Go workspaces for multi-module management
+- **Project Structure**: Multi-module workspace - `go.work`, `pkg/{module}/`, `cmd/`, `config/`
 - **Linting**: golangci-lint (configured in `.golangci.yml`)
 
 ### Required Tools
-- **go**: 1.24 or higher
+- **go**: 1.25 or higher (workspace support required)
 - **kubectl**: Kubernetes CLI for cluster interaction
 - **kind**: Local Kubernetes cluster for testing (or other local cluster)
 - **kubebuilder**: Optional - for regenerating CRDs and scaffolding
@@ -22,10 +22,14 @@
 - **kustomize**: Included with kubectl 1.14+, used for manifest management
 
 ### Dependency Management
-- **Go Modules**: `go.mod` and `go.sum` for dependency tracking
-- **Version Pinning**: Pin controller-runtime and client-go to compatible versions
-- **Vendor Directory**: Not used - rely on module cache
-- **Updating Dependencies**: Use `go get -u` carefully, test thoroughly after updates
+- **Per-Module Dependencies**: Each module (`cluster-handler`, `data-handler`, `resource-handler`) has its own `go.mod` and `go.sum`
+- **Version Pinning**: Pin controller-runtime and client-go to compatible versions in each module
+- **Go Workspaces (Local Only)**:
+  - Create `go.work` locally for easier multi-module development: `go work init ./pkg/cluster-handler ./pkg/data-handler ./pkg/resource-handler`
+  - `go.work` is in `.gitignore`
+  - **IMPORTANT**: `go.work` MUST NOT be committed - committing it would break the build and cause artifacts to ignore pinned versions
+  - Workspaces simplify local development by allowing cross-module references without publishing
+- **Updating Dependencies**: Update in each module separately with `go get -u`, test thoroughly after updates
 
 ## Coding Standards
 
@@ -64,15 +68,17 @@ Common markers:
 
 ### Testing Strategy
 
-**Unit Tests** (internal/resources, internal/webhook):
+**Unit Tests** (per module):
 - Test pure resource builder functions with table-driven tests
 - Mock nothing - builders are pure functions
 - Focus on correct Kubernetes manifest generation
+- Run tests in each module: `cd pkg/resource-handler && go test ./...`
 
-**Integration Tests** (internal/controller):
+**Integration Tests** (per module):
 - Use `envtest` - provides real Kubernetes API without full cluster
 - Test reconciliation loops end-to-end
 - Verify resource creation, updates, and status updates
+- Each module has its own integration tests scoped to its controllers
 
 **Test Commands**:
 ```bash
@@ -145,12 +151,13 @@ make manifests
 make install
 ```
 
-**Controller Changes** (`internal/controller/`):
-- Run `make test` frequently during development
+**Controller Changes** (`pkg/{module}/controller/`):
+- Run tests in the specific module: `cd pkg/resource-handler && go test ./...`
 - Integration tests catch reconciliation bugs early
 - Use `make run` for quick iteration (no docker build needed)
+- Changes in one module don't require rebuilding others during local development
 
-**Resource Builder Changes** (`internal/resources/`):
-- Write/update table-driven tests first
-- Run `make test` - very fast feedback loop
-- Integration tests verify end-to-end behavior
+**Module-Specific Development**:
+- **Resource Handler** (`pkg/resource-handler`): Component reconcilers and resource builders
+- **Cluster Handler** (`pkg/cluster-handler`): MultigresCluster orchestration logic
+- **Data Handler** (`pkg/data-handler`): Cell management and data plane configuration

docs/implementation-guide.md

@@ -0,0 +1,5 @@
+package multigrescluster
+
+func Dummy() string {
+	return "dummy string from cluster-handler's multigrescluster controller"
+}

internal/.gitkeep

@@ -0,0 +1,3 @@
+module github.com/numtide/multigres-operator/pkg/cluster-handler
+
+go 1.25.0

pkg/cluster-handler/controller/multigrescluster/dummy.go

@@ -0,0 +1,5 @@
+package cell
+
+func Dummy() string {
+	return "dummy string from data-handler's cell controller"
+}

pkg/cluster-handler/go.mod

@@ -0,0 +1,3 @@
+module github.com/numtide/multigres-operator/pkg/data-handler
+
+go 1.25.0