Skip to content

Commit 0c2c7c6

Browse files
clyang82clyang82
authored
Helm charts for maestro installation (#449)
* Migrate from OpenShift templates to Helm charts This commit replaces the OpenShift template-based deployment system with Helm charts for both maestro-server and maestro-agent components. Key changes: 1. **New Helm Charts** - Added charts/maestro-server/ with full Helm chart - Added charts/maestro-agent/ with full Helm chart - Both charts support MQTT, gRPC, and PubSub message brokers - Added optional embedded PostgreSQL and MQTT for development 2. **Makefile Refactoring** - Removed all oc process template generation code - Replaced template-based deploy/undeploy targets with Helm - Added simplified targets: deploy, deploy-dev, deploy-agent - Added utility targets: lint-charts, package-charts, template-server, template-agent - Updated help text to reflect Helm-based workflow 3. **E2E Test Infrastructure** - Updated test/setup/deploy_server.sh to use Helm - Updated test/setup/deploy_agent.sh to use Helm - Updated test/setup/env_setup.sh to remove template-based calls - E2E tests now deploy via Helm while maintaining same interface 4. **Documentation** - Added HELM_CHARTS.md with comprehensive guide - Added charts/README.md with migration guide - Added detailed READMEs for both charts - Documented all configuration options Benefits: - Cleaner deployment interface (make deploy, make deploy-dev) - Better configuration management via values.yaml - Support for all three message broker types - Embedded dependencies for development - Industry-standard Helm packaging Signed-off-by: $(git config user.name) <$(git config user.email)> Signed-off-by: clyang82 <chuyang@redhat.com> * modify to pass e2e tests * Add grpc broker support * remove templates Signed-off-by: clyang82 <chuyang@redhat.com> * Update to address comments Signed-off-by: clyang82 <chuyang@redhat.com> * test readme Signed-off-by: clyang82 <chuyang@redhat.com> * Add back grpc server tls support Signed-off-by: clyang82 <chuyang@redhat.com> * add grpc-client for testing Signed-off-by: clyang82 <chuyang@redhat.com> * Test rosa and gcp Signed-off-by: clyang82 <chuyang@redhat.com> * Address review comments Signed-off-by: clyang82 <chuyang@redhat.com> --------- Signed-off-by: $(git config user.name) <$(git config user.email)> Signed-off-by: clyang82 <chuyang@redhat.com>
1 parent 0c9847b commit 0c2c7c6

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

57 files changed

+2767
-4710
lines changed

‎Makefile‎

Lines changed: 72 additions & 157 deletions
Original file line numberDiff line numberDiff line change
@@ -20,11 +20,9 @@ version:=$(shell date +%s)
2020
# Tag for the image:
2121
image_tag ?= $(version)
2222

23-
# The namespace and the environment are calculated from the name of the user to
24-
# avoid clashes in shared infrastructure:
25-
environment:=${USER}
26-
namespace ?= maestro-${USER}
27-
agent_namespace ?= maestro-agent-${USER}
23+
# The namespace where maestro server and agent will be deployed.
24+
namespace ?= maestro
25+
agent_namespace ?= maestro-agent
2826

2927
# a tool for managing containers and images, etc. You can set it as docker
3028
container_tool ?= podman
@@ -53,7 +51,6 @@ db_port=5432
5351
db_user:=maestro
5452
db_password:=foobar-bizz-buzz
5553
db_password_file=${PWD}/secrets/db.password
56-
db_sslmode:=disable
5754
db_image?=quay.io/maestro/postgres:17.2
5855

5956
# Message broker connection details
@@ -62,59 +59,15 @@ mqtt_port ?= 1883
6259
mqtt_user ?= maestro
6360
mqtt_password_file ?= ${PWD}/secrets/mqtt.password
6461
mqtt_config_file ?= ${PWD}/secrets/mqtt.config
65-
mqtt_root_cert ?= ""
66-
mqtt_client_cert ?= ""
67-
mqtt_client_key ?= ""
6862

6963
# Log verbosity level
7064
klog_v:=2
7165

72-
# consumer name from the database. it is used by the maestro agent to identify itself
73-
consumer_name ?= cluster1
74-
75-
# Client id and secret are used to interact with other UHC services
76-
CLIENT_ID ?= maestro
77-
CLIENT_SECRET ?= maestro
78-
79-
# Enable gRPC server and disable gRPC broker by default
80-
ENABLE_GRPC_SERVER ?= true
81-
ENABLE_GRPC_BROKER ?= false
82-
83-
# Enable TLS
84-
ENABLE_TLS ?= false
85-
86-
# message driver type, mqtt or grpc, default is mqtt.
87-
MESSAGE_DRIVER_TYPE ?= mqtt
88-
89-
# default replicas for maestro server
90-
SERVER_REPLICAS ?= 1
91-
92-
# Enable set images
93-
POSTGRES_IMAGE ?= quay.io/maestro/postgres:17.2
94-
MQTT_IMAGE ?= quay.io/maestro/eclipse-mosquitto:2.0.18
95-
9666
# Test output files
9767
unit_test_json_output ?= ${PWD}/unit-test-results.json
9868
mqtt_integration_test_json_output ?= ${PWD}/mqtt-integration-test-results.json
9969
grpc_integration_test_json_output ?= ${PWD}/grpc-integration-test-results.json
10070

101-
# maestro services config
102-
maestro_svc_type ?= ClusterIP
103-
maestro_svc_node_port ?= 0
104-
grpc_svc_type ?= ClusterIP
105-
grpc_svc_node_port ?= 0
106-
107-
# maestro deployment config
108-
liveness_probe_init_delay_seconds ?= 15
109-
readiness_probe_init_delay_seconds ?= 20
110-
111-
# subscription config
112-
subscription_type ?= shared
113-
agent_topic ?= "\$$share/statussubscribers/sources/maestro/consumers/+/agentevents"
114-
115-
# default client certificate refresh/reload duration for message broker
116-
broker_client_cert_refresh_duration ?= 5m
117-
11871
# Prints a list of useful targets.
11972
help:
12073
@echo ""
@@ -131,16 +84,18 @@ help:
13184
@echo "make generate generate openapi modules"
13285
@echo "make image build docker image"
13386
@echo "make push push docker image"
134-
@echo "make deploy deploy via templates to local openshift instance"
135-
@echo "make undeploy undeploy from local openshift instance"
136-
@echo "make project create and use an Example project"
87+
@echo "make deploy deploy maestro server via Helm"
88+
@echo "make deploy-agent deploy maestro agent via Helm (requires consumer_name)"
89+
@echo "make undeploy undeploy maestro server"
90+
@echo "make undeploy-agent undeploy maestro agent"
91+
@echo "make lint-charts lint Helm charts"
13792
@echo "make clean delete temporary generated files"
13893
@echo "$(fake)"
13994
.PHONY: help
14095

14196
# Encourage consistent tool versions
14297
OPENAPI_GENERATOR_VERSION:=5.4.0
143-
GO_VERSION:=go1.24.
98+
GO_VERSION:=go1.25.
14499

145100
### Constants:
146101
version:=$(shell date +%s)
@@ -288,7 +243,6 @@ run/docs:
288243
clean:
289244
rm -rf \
290245
$(binary) \
291-
templates/*-template.json \
292246
data/generated/openapi/*.json \
293247
.PHONY: clean
294248

@@ -302,66 +256,6 @@ cmds:
302256
|| exit 1; \
303257
done
304258

305-
306-
# NOTE multiline variables are a PITA in Make. To use them in `oc process` later on, we need to first
307-
# export them as environment variables, then use the environment variable in `oc process`
308-
%-template:
309-
@if [ "$(ENABLE_TLS)" = "true" ]; then \
310-
TEMPLATE_FILE="templates/$*-tls-template.yml"; \
311-
else \
312-
TEMPLATE_FILE="templates/$*-template.yml"; \
313-
fi; \
314-
oc process \
315-
--filename="$$TEMPLATE_FILE" \
316-
--local="true" \
317-
--ignore-unknown-parameters="true" \
318-
--param="ENVIRONMENT=$(MAESTRO_ENV)" \
319-
--param="KLOG_V=$(klog_v)" \
320-
--param="SERVER_REPLICAS=$(SERVER_REPLICAS)" \
321-
--param="DATABASE_HOST=$(db_host)" \
322-
--param="DATABASE_NAME=$(db_name)" \
323-
--param="DATABASE_PASSWORD=$(db_password)" \
324-
--param="DATABASE_PORT=$(db_port)" \
325-
--param="DATABASE_USER=$(db_user)" \
326-
--param="DB_SSLMODE=$(db_sslmode)" \
327-
--param="POSTGRES_IMAGE=$(POSTGRES_IMAGE)" \
328-
--param="MQTT_HOST=$(mqtt_host)" \
329-
--param="MQTT_PORT=$(mqtt_port)" \
330-
--param="MQTT_USER=$(mqtt_user)" \
331-
--param="MQTT_PASSWORD=$(shell cat $(mqtt_password_file))" \
332-
--param="MQTT_ROOT_CERT=$(mqtt_root_cert)" \
333-
--param="MQTT_CLIENT_CERT=$(mqtt_client_cert)" \
334-
--param="MQTT_CLIENT_KEY=$(mqtt_client_key)" \
335-
--param="MQTT_IMAGE=$(MQTT_IMAGE)" \
336-
--param="IMAGE_REGISTRY=$(internal_image_registry)" \
337-
--param="IMAGE_REPOSITORY=$(image_repository)" \
338-
--param="IMAGE_TAG=$(image_tag)" \
339-
--param="VERSION=$(version)" \
340-
--param="AGENT_NAMESPACE=${agent_namespace}" \
341-
--param="EXTERNAL_APPS_DOMAIN=${external_apps_domain}" \
342-
--param="CONSUMER_NAME=$(consumer_name)" \
343-
--param="ENABLE_GRPC_SERVER=$(ENABLE_GRPC_SERVER)" \
344-
--param="MESSAGE_DRIVER_TYPE"=$(MESSAGE_DRIVER_TYPE) \
345-
--param="MAESTRO_SVC_TYPE"=$(maestro_svc_type) \
346-
--param="MAESTRO_SVC_NODE_PORT"=$(maestro_svc_node_port) \
347-
--param="GRPC_SVC_TYPE"=$(grpc_svc_type) \
348-
--param="GRPC_SVC_NODE_PORT"=$(grpc_svc_node_port) \
349-
--param="LIVENESS_PROBE_INIT_DELAY_SECONDS"=$(liveness_probe_init_delay_seconds) \
350-
--param="READINESS_PROBE_INIT_DELAY_SECONDS"=$(readiness_probe_init_delay_seconds) \
351-
--param="SUBSCRIPTION_TYPE"=$(subscription_type) \
352-
--param="AGENT_TOPIC"=$(agent_topic) \
353-
--param="BROKER_CLIENT_CERT_REFRESH_DURATION"=$(broker_client_cert_refresh_duration) \
354-
> "templates/$*-template.json"
355-
356-
357-
.PHONY: project
358-
project:
359-
$(oc) new-project "$(namespace)" || $(oc) project "$(namespace)" || true
360-
361-
.PHONY: agent-project
362-
agent-project:
363-
$(oc) new-project "$(agent_namespace)" || $(oc) project "$(agent_namespace)" || true
364-
365259
.PHONY: image
366260
image: cmds
367261
$(container_tool) build -t "$(external_image_registry)/$(image_repository):$(image_tag)" .
@@ -371,50 +265,44 @@ e2e-image:
371265
$(container_tool) build -f Dockerfile.e2e -t "$(external_image_registry)/$(image_repository)-e2e:$(image_tag)" .
372266

373267
.PHONY: push
374-
push: image project
268+
push: image
375269
$(container_tool) push "$(external_image_registry)/$(image_repository):$(image_tag)"
376270

377-
deploy-%: project %-template
378-
$(oc) apply -n $(namespace) --filename="templates/$*-template.json" | egrep --color=auto 'configured|$$'
379-
380-
undeploy-%: project %-template
381-
$(oc) delete -n $(namespace) --filename="templates/$*-template.json" | egrep --color=auto 'deleted|$$'
271+
# Deploy Maestro server using Helm charts
272+
.PHONY: deploy
273+
deploy:
274+
helm upgrade --install maestro-server \
275+
./charts/maestro-server \
276+
--namespace $(namespace) \
277+
--create-namespace \
278+
--set mqtt.enabled=true \
279+
--set route.enabled=true \
280+
--set postgresql.enabled=true
281+
282+
# Undeploy Maestro server using Helm charts
283+
.PHONY: undeploy
284+
undeploy:
285+
helm uninstall maestro-server --namespace $(namespace) || true
382286

287+
# Deploy Maestro agent using Helm charts
288+
# Optional: Set install_work_crds=true to install CRDs (default: false to skip if already exists)
383289
.PHONY: deploy-agent
384-
deploy-agent: agent-project agent-template
385-
$(oc) apply -n $(agent_namespace) --filename="templates/agent-template.json" | egrep --color=auto 'configured|$$'
386-
290+
deploy-agent:
291+
@if [ -z "$(consumer_name)" ]; then \
292+
echo "Error: consumer_name must be set"; \
293+
exit 1; \
294+
fi
295+
helm upgrade --install maestro-agent \
296+
./charts/maestro-agent \
297+
--namespace $(agent_namespace) \
298+
--create-namespace \
299+
--set consumerName=$(consumer_name) \
300+
--set installWorkCRDs=$(if $(install_work_crds),$(install_work_crds),false)
301+
302+
# Undeploy Maestro agent using Helm charts
387303
.PHONY: undeploy-agent
388-
undeploy-agent: agent-project agent-template
389-
$(oc) delete -n $(agent_namespace) --filename="templates/agent-template.json" | egrep --color=auto 'deleted|$$'
390-
391-
.PHONY: template
392-
template: \
393-
db-template \
394-
mqtt-template \
395-
service-template \
396-
route-template \
397-
$(NULL)
398-
399-
# Depending on `template` first helps clustering the "foo configured", "bar unchanged",
400-
# "baz deleted" messages at the end, after all the noisy templating.
401-
.PHONY: deploy
402-
deploy: \
403-
template \
404-
deploy-db \
405-
deploy-mqtt \
406-
deploy-service \
407-
deploy-route \
408-
$(NULL)
409-
410-
.PHONY: undeploy
411-
undeploy: \
412-
template \
413-
undeploy-db \
414-
undeploy-mqtt \
415-
undeploy-service \
416-
undeploy-route \
417-
$(NULL)
304+
undeploy-agent:
305+
helm uninstall maestro-agent --namespace $(agent_namespace) || true
418306

419307
.PHONY: db/setup
420308
db/setup:
@@ -457,12 +345,12 @@ test-env/setup:
457345
./test/setup/env_setup.sh
458346
.PHONY: test-env/setup
459347

460-
# Deploy the Maestro server component to the test environment
348+
# Deploy the Maestro server component to the test environment using Helm
461349
test-env/deploy-server:
462350
./test/setup/deploy_server.sh
463351
.PHONY: test-env/deploy-server
464352

465-
# Deploy the Maestro agent component to the test environment
353+
# Deploy the Maestro agent component to the test environment using Helm
466354
# Configures agent to connect to the deployed server
467355
test-env/deploy-agent:
468356
./test/setup/deploy_agent.sh
@@ -474,7 +362,7 @@ test-env/cleanup:
474362
./test/setup/env_cleanup.sh
475363
.PHONY: test-env/cleanup
476364

477-
# Prepare the test environment
365+
# Prepare the test environment using Helm charts
478366
test-env: test-env/cleanup test-env/setup test-env/deploy-server test-env/deploy-agent
479367
.PHONY: test-env
480368

@@ -505,6 +393,7 @@ e2e-test/run:
505393
# Example:
506394
# make e2e-test
507395
# ENABLE_MAESTRO_TLS=true make e2e-test
396+
# NOTE: Uses Helm charts for deployment
508397
e2e-test: test-env e2e-test/run
509398
.PHONY: e2e-test
510399

@@ -522,3 +411,29 @@ endif
522411
upgrade-test: test-env/cleanup test-env/setup
523412
./test/upgrade/test.sh
524413
.PHONY: upgrade-test
414+
415+
# ==============================================================================
416+
# Helm Chart Utility Targets
417+
# ==============================================================================
418+
419+
# Lint all Helm charts
420+
lint-charts:
421+
helm lint charts/maestro-server
422+
helm lint charts/maestro-agent
423+
.PHONY: lint-charts
424+
425+
# Package all Helm charts
426+
package-charts:
427+
helm package charts/maestro-server -d charts/
428+
helm package charts/maestro-agent -d charts/
429+
.PHONY: package-charts
430+
431+
# Render maestro-server chart templates (dry-run)
432+
template-server:
433+
helm template maestro-server ./charts/maestro-server --namespace $(namespace)
434+
.PHONY: template-server
435+
436+
# Render maestro-agent chart templates (dry-run)
437+
template-agent:
438+
helm template maestro-agent ./charts/maestro-agent --namespace $(agent_namespace)
439+
.PHONY: template-agent

‎README.md‎

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -286,15 +286,12 @@ $ export external_apps_domain=`oc -n openshift-ingress-operator get ingresscontr
286286

287287
If you want to push the image to your OpenShift cluster default registry and then deploy it to the cluster. You need to follow [this document](https://docs.openshift.com/container-platform/4.13/registry/securing-exposing-registry.html) to expose a default registry manually and login into the registry with podman. Then run `make push` to push the image to the registry.
288288

289-
If you want to use the existing image, set the image environment variables.
289+
If you want to use the default image, you can skip the `make push` step.
290290

291291
```shell
292-
$ export internal_image_registry=quay.io/redhat-user-workloads/maestro-rhtap-tenant
293-
$ export image_repository=maestro/maestro
294-
$ export image_tag=latest
295292
$ make deploy
296293

297-
$ oc get pod -n "maestro-$USER"
294+
$ oc get pod -n maestro
298295
NAME READY STATUS RESTARTS AGE
299296
maestro-85c847764-4xdt6 1/1 Running 0 62s
300297
maestro-db-5d4c4679f5-r92vg 1/1 Running 0 61s
@@ -306,7 +303,7 @@ maestro-mqtt-6cb7bdf46c-kcczm 1/1 Running 0 63s
306303
```shell
307304
$ curl -k -X POST -H "Authorization: Bearer $TOKEN" \
308305
-H "Content-Type: application/json" \
309-
https://maestro.${external_apps_domain}/api/maestro/v1/consumers \
306+
https://maestro-maestro.${external_apps_domain}/api/maestro/v1/consumers \
310307
-d '{
311308
"name": "cluster1"
312309
}'
@@ -328,8 +325,9 @@ You should get a response like this:
328325

329326
```shell
330327
$ export consumer_name=cluster1
328+
$ export install_work_crds=false
331329
$ make deploy-agent
332-
$ oc get pod -n "maestro-agent-$USER"
330+
$ oc get pod -n maestro-agent
333331
NAME READY STATUS RESTARTS AGE
334332
maestro-agent-5dc9f5b4bf-8jcvq 1/1 Running 0 13s
335333
```

‎charts/README.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Maestro Helm Charts
2+
3+
This directory contains Helm charts for deploying Maestro components.
4+
5+
## Available Charts
6+
7+
### maestro-server
8+
The Maestro Server chart deploys the main server component that:
9+
- Stores resources and their status in a database
10+
- Sends resources to message brokers via CloudEvents
11+
- Provides REST and gRPC APIs
12+
13+
[maestro-server Documentation](./maestro-server/README.md)
14+
15+
### maestro-agent
16+
The Maestro Agent chart deploys the agent component that:
17+
- Receives resources from the server via CloudEvents
18+
- Applies resources to the target cluster
19+
- Reports back resource status
20+
21+
[maestro-agent Documentation](./maestro-agent/README.md)
22+

0 commit comments

Comments
 (0)