|
2 | 2 |
|
3 | 3 | [](https://github.com/fluxcd/source-watcher/actions) |
4 | 4 |
|
5 | | -Example consumer of the GitOps Toolkit Source APIs. |
| 5 | +A Flux extension controller that enables advanced source composition and decomposition patterns for GitOps workflows. |
6 | 6 |
|
7 | | - |
| 7 | +## Overview |
8 | 8 |
|
9 | | -## Guides |
| 9 | +source-watcher introduces the **ArtifactGenerator** API, which allows you to: |
10 | 10 |
|
11 | | -* [Watching for source changes](https://fluxcd.io/flux/gitops-toolkit/source-watcher/) |
| 11 | +- 🔗 **Compose** multiple Flux sources (GitRepository, OCIRepository, Bucket) into a single deployable artifact |
| 12 | +- 📦 **Decompose** monorepos into multiple independent artifacts with separate deployment lifecycles |
| 13 | +- 🎯 **Optimize** reconciliation by only triggering updates when specific paths change |
| 14 | +- 🏗️ **Structure** complex deployments from distributed sources maintained by different teams |
| 15 | + |
| 16 | +## Quick Start |
| 17 | + |
| 18 | +### Prerequisites |
| 19 | + |
| 20 | +- Flux v2.7.0 or later |
| 21 | +- source-watcher v2.0.0 or later |
| 22 | +- kustomize-controller & helm-controller with `--feature-gates=ExternalArtifacts` |
| 23 | + |
| 24 | +### Example: Composing Multiple Sources |
| 25 | + |
| 26 | +Deploy an application that combines Kubernetes manifests from different sources: |
| 27 | + |
| 28 | +```yaml |
| 29 | +apiVersion: source.extensions.fluxcd.io/v1beta1 |
| 30 | +kind: ArtifactGenerator |
| 31 | +metadata: |
| 32 | + name: my-app |
| 33 | + namespace: flux-system |
| 34 | +spec: |
| 35 | + sources: |
| 36 | + - alias: backend |
| 37 | + kind: GitRepository |
| 38 | + name: backend-repo |
| 39 | + - alias: frontend |
| 40 | + kind: OCIRepository |
| 41 | + name: frontend-oci |
| 42 | + - alias: config |
| 43 | + kind: Bucket |
| 44 | + name: config-bucket |
| 45 | + artifacts: |
| 46 | + - name: my-app-composite |
| 47 | + copy: |
| 48 | + - from: "@backend/deploy/k8s/**" |
| 49 | + to: "@artifact/backend/" |
| 50 | + - from: "@frontend/k8s/manifests/*.yaml" |
| 51 | + to: "@artifact/frontend/" |
| 52 | + - from: "@config/prod/settings.yaml" |
| 53 | + to: "@artifact/config.yaml" |
| 54 | +--- |
| 55 | +apiVersion: kustomize.toolkit.fluxcd.io/v1 |
| 56 | +kind: Kustomization |
| 57 | +metadata: |
| 58 | + name: my-app |
| 59 | + namespace: flux-system |
| 60 | +spec: |
| 61 | + interval: 10m |
| 62 | + sourceRef: |
| 63 | + kind: ExternalArtifact |
| 64 | + name: my-app-composite |
| 65 | + path: "./" |
| 66 | + prune: true |
| 67 | +``` |
| 68 | +
|
| 69 | +### Example: Monorepo Decomposition |
| 70 | +
|
| 71 | +Split a monorepo into independently deployable services: |
| 72 | +
|
| 73 | +```yaml |
| 74 | +apiVersion: source.extensions.fluxcd.io/v1beta1 |
| 75 | +kind: ArtifactGenerator |
| 76 | +metadata: |
| 77 | + name: platform-services |
| 78 | + namespace: flux-system |
| 79 | +spec: |
| 80 | + sources: |
| 81 | + - alias: monorepo |
| 82 | + kind: GitRepository |
| 83 | + name: platform-monorepo |
| 84 | + artifacts: |
| 85 | + - name: policy-config |
| 86 | + copy: |
| 87 | + - from: "@monorepo/platform/policy/**" |
| 88 | + to: "@artifact/" |
| 89 | + - name: auth-service |
| 90 | + copy: |
| 91 | + - from: "@monorepo/services/auth/deploy/**" |
| 92 | + to: "@artifact/" |
| 93 | + - name: api-gateway |
| 94 | + copy: |
| 95 | + - from: "@monorepo/services/gateway/deploy/**" |
| 96 | + to: "@artifact/" |
| 97 | + exclude: ["**/charts/**"] |
| 98 | +``` |
| 99 | +
|
| 100 | +Each service gets its own ExternalArtifact with an independent revision. |
| 101 | +Changes to `services/auth/` only trigger reconciliation of the auth-service, |
| 102 | +leaving other services untouched. |
| 103 | + |
| 104 | +### Example: Helm Chart Composition |
| 105 | + |
| 106 | +Compose Helm charts with custom values from different sources: |
| 107 | + |
| 108 | +```yaml |
| 109 | +apiVersion: source.extensions.fluxcd.io/v1beta1 |
| 110 | +kind: ArtifactGenerator |
| 111 | +metadata: |
| 112 | + name: podinfo |
| 113 | + namespace: apps |
| 114 | +spec: |
| 115 | + sources: |
| 116 | + - alias: chart |
| 117 | + kind: OCIRepository |
| 118 | + name: podinfo-chart |
| 119 | + namespace: apps |
| 120 | + - alias: repo |
| 121 | + kind: GitRepository |
| 122 | + name: podinfo-values |
| 123 | + namespace: apps |
| 124 | + artifacts: |
| 125 | + - name: podinfo-composite |
| 126 | + originRevision: "@chart" # Track chart version |
| 127 | + copy: |
| 128 | + - from: "@chart/" |
| 129 | + to: "@artifact/" |
| 130 | + - from: "@repo/charts/podinfo/values-prod.yaml" |
| 131 | + to: "@artifact/podinfo/values.yaml" # Override default values |
| 132 | +``` |
| 133 | + |
| 134 | +## Key Features |
| 135 | + |
| 136 | +### Content-based Revision Tracking |
| 137 | + |
| 138 | +The generated artifacts are versioned as `latest@sha256:<hash>` where the hash |
| 139 | +is computed from the artifact's content. This means: |
| 140 | + |
| 141 | +- ✅ Identical content = same revision (no unnecessary reconciliations) |
| 142 | +- ✅ Any change = new revision (guaranteed updates) |
| 143 | +- ✅ Automatic change detection across all source types |
| 144 | + |
| 145 | +The controller attaches the origin source revision (e.g. Git commit SHA) |
| 146 | +to the generated artifact metadata for traceability and Flux commit status updates. |
| 147 | + |
| 148 | +### Flexible Copy Operations |
| 149 | + |
| 150 | +Copy operations support glob patterns and exclusions: |
| 151 | + |
| 152 | +- `@source/file.yaml` → `@artifact/dest/` - Copy file into directory |
| 153 | +- `@source/dir/` → `@artifact/dest/` - Copy directory as subdirectory |
| 154 | +- `@source/dir/**` → `@artifact/dest/` - Copy directory contents |
| 155 | +- `@source/file.yaml` → `@artifact/existing.yaml` - Later copy overwrites earlier ones |
| 156 | + |
| 157 | +## Use Cases |
| 158 | + |
| 159 | +- **Multi-Team Collaboration** - Different teams maintain different Kubernetes |
| 160 | + configurations in separate repositories. ArtifactGenerator combines them into deployable |
| 161 | + units while maintaining clear ownership boundaries. |
| 162 | +- **Environment-Specific Composition** Compose base manifests with environment-specific |
| 163 | + configurations from different sources without duplicating files across repositories. |
| 164 | +- **Vendor Chart Customization** - Combine upstream Helm charts from OCI registries with |
| 165 | + your organization's custom values and configuration overrides stored in Git. |
| 166 | +- **Selective Deployment** - Deploy only changed components from large repositories |
| 167 | + by decomposing them into focused artifacts. |
| 168 | + |
| 169 | +## API Reference |
| 170 | + |
| 171 | +- [ArtifactGenerator CRD](docs/spec/v1beta1/artifactgenerators.md) |
| 172 | + |
| 173 | +## Contributing |
| 174 | + |
| 175 | +This project is Apache 2.0 licensed and accepts contributions via GitHub pull requests. |
| 176 | +To start contributing please see the [development guide](CONTRIBUTING.md). |
0 commit comments