Skip to content

Commit d744e99

Browse files
mwennrichmwennrich
authored
feat: add envoy gateway to service clusters (for isolated clusters) (#522)
* feat: add envoy gateway to service clusters (for isolated clusters)
1 parent 6f0c598 commit d744e99

File tree

10 files changed

+287
-7
lines changed

10 files changed

+287
-7
lines changed

‎control-plane/roles/isolated-clusters/README.md‎

Lines changed: 26 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,26 @@ It contains the services:
88
- Registry
99
- DNS
1010
- NTP
11+
- Ingress Controller (nginx-ingress)
12+
- Envoy Gateway (optional, alternative to nginx-ingress)
1113

1214
A cert-manager is expected to be deployed by the user beforehand.
1315

16+
## Ingress vs Envoy Gateway
17+
18+
This role supports two options for exposing services:
19+
20+
1. **nginx-ingress** (default, enabled by default): Traditional Ingress controller for HTTP/HTTPS traffic and TCP/UDP proxying
21+
2. **Envoy Gateway** (optional): Modern Gateway API implementation supporting HTTP/HTTPS, TCP, and UDP protocols
22+
23+
You can enable/disable each option independently using the `isolated_clusters_ingress_controller_enabled` and `isolated_clusters_envoy_gateway_enabled` variables. Both can be enabled simultaneously if needed.
24+
1425
## Variables
1526

1627
The `control-plane-defaults` folder contains defaults that are used by multiple roles in the control-plane directory. You can look up all the default values [here](control-plane-defaults/main.yaml).
1728

29+
**Note:** Variables marked with `yes*` are conditionally mandatory - they are only required when the corresponding feature is enabled.
30+
1831
| Name | Mandatory | Description |
1932
| ---------------------------------------------------------------- | --------- | ------------------------------------------------------------------------------------------------ |
2033
| isolated_clusters_virtual_garden_kubeconfig | | The kubeconfig to access the virtual garden as a string value. |
@@ -24,11 +37,20 @@ The `control-plane-defaults` folder contains defaults that are used by multiple
2437
| isolated_clusters_dns_image_name | | The image name of the dns service for the partition. |
2538
| isolated_clusters_dns_image_tag | yes | The tag or version of the dns service container image. |
2639
| isolated_clusters_dns_namespace | | The namespace to deploy the dns server to. |
40+
| isolated_clusters_ingress_controller_enabled | | Enable/disable nginx-ingress controller deployment (default: true). |
2741
| isolated_clusters_ingress_controller_namespace | | The namespace where the ingress controller should be deployed to. |
28-
| isolated_clusters_ingress_controller_chart_version | yes | The version of the ingress controller chart. |
42+
| isolated_clusters_ingress_controller_chart_version | yes* | The version of the ingress controller chart (required if ingress enabled). |
2943
| isolated_clusters_ingress_controller_chroot | | Indicates if the image should have a changed root. |
30-
| isolated_clusters_ingress_controller_load_balancer_ip | yes | The load balancer source ip of the ingress controller. |
31-
| isolated_clusters_ingress_controller_load_balancer_source_ranges | yes | The load balancer source ranges of the ingress controller. |
44+
| isolated_clusters_ingress_controller_load_balancer_ip | yes* | The load balancer source ip of the ingress controller (required if ingress enabled). |
45+
| isolated_clusters_ingress_controller_load_balancer_source_ranges | yes* | The load balancer source ranges of the ingress controller (required if ingress enabled). |
46+
| isolated_clusters_envoy_gateway_enabled | | Enable/disable Envoy Gateway deployment (default: false). |
47+
| isolated_clusters_envoy_gateway_namespace | | The namespace where Envoy Gateway should be deployed to. |
48+
| isolated_clusters_envoy_gateway_chart_version | yes* | The version of the Envoy Gateway helm chart (required if envoy gateway enabled). |
49+
| isolated_clusters_envoy_gateway_replicas | | Number of Envoy Gateway replicas (default: 1). |
50+
| isolated_clusters_envoy_gateway_load_balancer_ip | | The load balancer IP for the Envoy Gateway service (optional). |
51+
| isolated_clusters_envoy_gateway_load_balancer_source_ranges | | The load balancer source ranges for Envoy Gateway (optional). |
52+
| isolated_clusters_envoy_gateway_registry_tls_secret_name | | The name of the TLS secret for registry HTTPS listener in Envoy Gateway. |
53+
| isolated_clusters_registry_fqdn | yes* | The FQDN for the registry service (required if envoy gateway enabled). |
3254
| isolated_clusters_registry_image_name | | The image name of the registry service for the partition. |
3355
| isolated_clusters_registry_image_tag | yes | The tag or version of the registry service container image. |
3456
| isolated_clusters_registry_namespace | | The namespace for the registry used for isolated clusters. |
@@ -37,6 +59,6 @@ The `control-plane-defaults` folder contains defaults that are used by multiple
3759
| isolated_clusters_registry_oci_mirror_config | | Contains a mapping of source and destination images for specific versions. |
3860
| isolated_clusters_registry_storage_size | | The storage size of the registry used for isolated clusters. |
3961
| isolated_clusters_registry_storage_class_name | | The storageClassName of the registry used for isolated clusters. |
40-
| isolated_clusters_registry_ingress_fqdn | yes | The full name of the registry used for isolated clusters. |
62+
| isolated_clusters_registry_ingress_fqdn | yes* | The full name of the registry ingress (required if ingress enabled). |
4163
| isolated_clusters_registry_ingress_annotations | | Optional ingress annotations for the registry used for isolated clusters. |
4264
| isolated_clusters_partition_services_cluster | | The cluster to deploy the services like ntp, dns, ingress, cert manager and the OCI registry to. |

‎control-plane/roles/isolated-clusters/defaults/main/main.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,12 +9,21 @@ isolated_clusters_dns_image_name: coredns/coredns
99
isolated_clusters_dns_image_tag:
1010
isolated_clusters_dns_namespace: dns-service
1111

12+
isolated_clusters_ingress_controller_enabled: true
1213
isolated_clusters_ingress_controller_namespace: ingress
1314
isolated_clusters_ingress_controller_chart_version:
1415
isolated_clusters_ingress_controller_chroot: true
1516
isolated_clusters_ingress_controller_load_balancer_ip:
1617
isolated_clusters_ingress_controller_load_balancer_source_ranges:
1718

19+
isolated_clusters_envoy_gateway_enabled: false
20+
isolated_clusters_envoy_gateway_namespace: envoy-gateway-system
21+
isolated_clusters_envoy_gateway_chart_version:
22+
isolated_clusters_envoy_gateway_replicas: 1
23+
isolated_clusters_envoy_gateway_load_balancer_ip:
24+
isolated_clusters_envoy_gateway_load_balancer_source_ranges:
25+
isolated_clusters_envoy_gateway_registry_tls_secret_name: registry-tls-cert
26+
1827
isolated_clusters_group_label: "{{ metal_control_plane_stage_name }}"
1928
isolated_clusters_partition_services_cluster: []
2029
# - name: somename

‎control-plane/roles/isolated-clusters/tasks/envoy-gateway.yaml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
---
2+
- name: Create namespace for envoy-gateway
3+
k8s:
4+
definition:
5+
apiVersion: v1
6+
kind: Namespace
7+
metadata:
8+
name: "{{ isolated_clusters_envoy_gateway_namespace }}"
9+
labels:
10+
app.kubernetes.io/part-of: "{{ isolated_clusters_group_label }}"
11+
apply: yes
12+
kubeconfig: "{{ cluster.kubeconfig }}"
13+
14+
- name: Deploy envoy gateway
15+
kubernetes.core.helm:
16+
name: envoy-gateway
17+
chart_ref: oci://docker.io/envoyproxy/gateway-helm
18+
chart_version: "{{ isolated_clusters_envoy_gateway_chart_version }}"
19+
namespace: "{{ isolated_clusters_envoy_gateway_namespace }}"
20+
values: "{{ lookup('template', 'envoy-gateway/values.yaml') | from_yaml }}"
21+
kubeconfig: "{{ cluster.kubeconfig }}"
22+
23+
- name: Wait for envoy-gateway deployment to be available
24+
kubernetes.core.k8s_info:
25+
kind: Deployment
26+
name: envoy-gateway
27+
namespace: "{{ isolated_clusters_envoy_gateway_namespace }}"
28+
kubeconfig: "{{ cluster.kubeconfig }}"
29+
wait: yes
30+
wait_condition:
31+
type: Available
32+
status: "True"
33+
wait_timeout: 300
34+
35+
- name: Deploy gateway resources
36+
k8s:
37+
definition: "{{ lookup('template', 'envoy-gateway/gateway.yaml') }}"
38+
apply: yes
39+
kubeconfig: "{{ cluster.kubeconfig }}"
40+
41+
- name: Deploy gateway routes
42+
k8s:
43+
definition: "{{ lookup('template', 'envoy-gateway/routes.yaml') }}"
44+
apply: yes
45+
kubeconfig: "{{ cluster.kubeconfig }}"

‎control-plane/roles/isolated-clusters/tasks/main.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,26 @@
1212
- isolated_clusters_registry_image_tag is not none
1313
- isolated_clusters_registry_oci_mirror_image_name is not none
1414
- isolated_clusters_registry_oci_mirror_image_tag is not none
15+
16+
- name: Check mandatory variables for ingress controller
17+
assert:
18+
fail_msg: "not all mandatory ingress controller variables given, check role documentation"
19+
quiet: yes
20+
that:
1521
- isolated_clusters_ingress_controller_chart_version is not none
1622
- isolated_clusters_ingress_controller_load_balancer_ip is not none
1723
- isolated_clusters_ingress_controller_load_balancer_source_ranges is not none
1824
- isolated_clusters_registry_ingress_fqdn is not none
25+
when: isolated_clusters_ingress_controller_enabled | bool
26+
27+
- name: Check mandatory variables for envoy gateway
28+
assert:
29+
fail_msg: "not all mandatory envoy gateway variables given, check role documentation"
30+
quiet: yes
31+
that:
32+
- isolated_clusters_envoy_gateway_chart_version is not none
33+
- isolated_clusters_registry_fqdn is not none
34+
when: isolated_clusters_envoy_gateway_enabled | bool
1935

2036
- name: Loop over clusters to install services
2137
include_tasks: services.yaml

‎control-plane/roles/isolated-clusters/tasks/ntp.yaml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,6 @@
22
- name: Create namespace {{ isolated_clusters_ntp_namespace }}
33
k8s:
44
definition:
5-
api_version: v1
65
kind: Namespace
76
metadata:
87
name: "{{ isolated_clusters_ntp_namespace }}"

‎control-plane/roles/isolated-clusters/tasks/registry.yaml‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,6 @@
22
- name: Create namespace {{ isolated_clusters_registry_namespace }}
33
k8s:
44
definition:
5-
api_version: v1
65
kind: Namespace
76
metadata:
87
name: "{{ isolated_clusters_registry_namespace }}"
@@ -20,5 +19,12 @@
2019
loop:
2120
- registry-sts.yaml
2221
- registry-service.yaml
23-
- registry-ingress.yaml
2422
- registry-oci-mirror.yaml
23+
24+
- name: Deploy registry ingress
25+
k8s:
26+
definition: "{{ lookup('template', 'registry/registry-ingress.yaml') }}"
27+
namespace: "{{ isolated_clusters_registry_namespace }}"
28+
kubeconfig: "{{ cluster.kubeconfig }}"
29+
apply: yes
30+
when: isolated_clusters_ingress_controller_enabled | bool

‎control-plane/roles/isolated-clusters/tasks/services.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33

44
- name: Install nginx-ingress
55
import_tasks: ingress.yaml
6+
when: isolated_clusters_ingress_controller_enabled | bool
67

78
- name: Install ntp
89
import_tasks: ntp.yaml
@@ -12,3 +13,7 @@
1213

1314
- name: Install deployment
1415
import_tasks: registry.yaml
16+
17+
- name: Install envoy-gateway
18+
import_tasks: envoy-gateway.yaml
19+
when: isolated_clusters_envoy_gateway_enabled | bool

‎control-plane/roles/isolated-clusters/templates/envoy-gateway/gateway.yaml‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
---
2+
apiVersion: gateway.envoyproxy.io/v1alpha1
3+
kind: EnvoyProxy
4+
metadata:
5+
name: custom-proxy-config
6+
namespace: {{ isolated_clusters_envoy_gateway_namespace }}
7+
spec:
8+
provider:
9+
type: Kubernetes
10+
kubernetes:
11+
envoyService:
12+
type: LoadBalancer
13+
externalTrafficPolicy: "Local"
14+
loadBalancerIP: "{{ isolated_clusters_envoy_gateway_load_balancer_ip }}"
15+
loadBalancerSourceRanges:
16+
{{ isolated_clusters_envoy_gateway_load_balancer_source_ranges | to_nice_yaml | indent(10, true) }}
17+
---
18+
apiVersion: gateway.networking.k8s.io/v1
19+
kind: GatewayClass
20+
metadata:
21+
name: envoy-gateway-class
22+
spec:
23+
controllerName: gateway.envoyproxy.io/gatewayclass-controller
24+
parametersRef:
25+
group: gateway.envoyproxy.io
26+
kind: EnvoyProxy
27+
name: custom-proxy-config
28+
namespace: {{ isolated_clusters_envoy_gateway_namespace }}
29+
---
30+
apiVersion: gateway.networking.k8s.io/v1
31+
kind: Gateway
32+
metadata:
33+
name: main-gateway
34+
namespace: {{ isolated_clusters_envoy_gateway_namespace }}
35+
labels:
36+
app.kubernetes.io/part-of: "{{ isolated_clusters_group_label }}"
37+
spec:
38+
gatewayClassName: envoy-gateway-class
39+
listeners:
40+
- name: https-registry
41+
protocol: HTTPS
42+
port: 443
43+
hostname: "{{ isolated_clusters_registry_fqdn }}"
44+
tls:
45+
mode: Terminate
46+
certificateRefs:
47+
- name: {{ isolated_clusters_envoy_gateway_registry_tls_secret_name }}
48+
namespace: {{ isolated_clusters_registry_namespace }}
49+
allowedRoutes:
50+
namespaces:
51+
from: Selector
52+
selector:
53+
matchLabels:
54+
kubernetes.io/metadata.name: {{ isolated_clusters_registry_namespace }}
55+
56+
- name: dns-tcp
57+
protocol: TCP
58+
port: 53
59+
allowedRoutes:
60+
namespaces:
61+
from: Selector
62+
selector:
63+
matchLabels:
64+
kubernetes.io/metadata.name: {{ isolated_clusters_dns_namespace }}
65+
- name: dns-udp
66+
protocol: UDP
67+
port: 53
68+
allowedRoutes:
69+
namespaces:
70+
from: Selector
71+
selector:
72+
matchLabels:
73+
kubernetes.io/metadata.name: {{ isolated_clusters_dns_namespace }}
74+
75+
- name: ntp-udp
76+
protocol: UDP
77+
port: 123
78+
allowedRoutes:
79+
namespaces:
80+
from: Selector
81+
selector:
82+
matchLabels:
83+
kubernetes.io/metadata.name: {{ isolated_clusters_ntp_namespace }}
84+
---
85+
apiVersion: gateway.networking.k8s.io/v1beta1
86+
kind: ReferenceGrant
87+
metadata:
88+
name: allow-gateway-to-read-tls-secret
89+
namespace: {{ isolated_clusters_registry_namespace }}
90+
labels:
91+
app.kubernetes.io/part-of: "{{ isolated_clusters_group_label }}"
92+
spec:
93+
from:
94+
- group: gateway.networking.k8s.io
95+
kind: Gateway
96+
namespace: {{ isolated_clusters_envoy_gateway_namespace }}
97+
to:
98+
- group: ""
99+
kind: Secret
100+
name: {{ isolated_clusters_envoy_gateway_registry_tls_secret_name }}

‎control-plane/roles/isolated-clusters/templates/envoy-gateway/routes.yaml‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
---
2+
apiVersion: gateway.networking.k8s.io/v1
3+
kind: HTTPRoute
4+
metadata:
5+
name: registry-mirror-route
6+
namespace: {{ isolated_clusters_registry_namespace }}
7+
labels:
8+
app.kubernetes.io/part-of: "{{ isolated_clusters_group_label }}"
9+
spec:
10+
parentRefs:
11+
- name: main-gateway
12+
namespace: {{ isolated_clusters_envoy_gateway_namespace }}
13+
sectionName: https-registry
14+
hostnames:
15+
- "{{ isolated_clusters_registry_fqdn }}"
16+
rules:
17+
- matches:
18+
- method: GET
19+
- method: HEAD
20+
backendRefs:
21+
- name: registry
22+
port: 5000
23+
24+
---
25+
apiVersion: gateway.networking.k8s.io/v1alpha2
26+
kind: TCPRoute
27+
metadata:
28+
name: dns-tcp-route
29+
namespace: {{ isolated_clusters_dns_namespace }}
30+
labels:
31+
app.kubernetes.io/part-of: "{{ isolated_clusters_group_label }}"
32+
spec:
33+
parentRefs:
34+
- name: main-gateway
35+
namespace: {{ isolated_clusters_envoy_gateway_namespace }}
36+
sectionName: dns-tcp
37+
rules:
38+
- backendRefs:
39+
- name: coredns-tcp
40+
port: 53
41+
42+
---
43+
apiVersion: gateway.networking.k8s.io/v1alpha2
44+
kind: UDPRoute
45+
metadata:
46+
name: dns-udp-route
47+
namespace: {{ isolated_clusters_dns_namespace }}
48+
labels:
49+
app.kubernetes.io/part-of: "{{ isolated_clusters_group_label }}"
50+
spec:
51+
parentRefs:
52+
- name: main-gateway
53+
namespace: {{ isolated_clusters_envoy_gateway_namespace }}
54+
sectionName: dns-udp
55+
rules:
56+
- backendRefs:
57+
- name: coredns-udp
58+
port: 53
59+
60+
---
61+
apiVersion: gateway.networking.k8s.io/v1alpha2
62+
kind: UDPRoute
63+
metadata:
64+
name: ntp-udp-route
65+
namespace: {{ isolated_clusters_ntp_namespace }}
66+
labels:
67+
app.kubernetes.io/part-of: "{{ isolated_clusters_group_label }}"
68+
spec:
69+
parentRefs:
70+
- name: main-gateway
71+
namespace: {{ isolated_clusters_envoy_gateway_namespace }}
72+
sectionName: ntp-udp
73+
rules:
74+
- backendRefs:
75+
- name: chrony
76+
port: 123

‎control-plane/roles/isolated-clusters/templates/envoy-gateway/values.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
deployment:
2+
replicas: {{ isolated_clusters_envoy_gateway_replicas | default(1) }}

0 commit comments

Comments
 (0)