Skip to content

Commit 5372f6a

Browse files
Enable ServiceMonitor to use cert-manager-managed serving-cert with TLS verification
Adds a patch to configure ServiceMonitor with to ensure TLS verification using cert-manager certificates. Updates documentation and corrects misaligned comments.
1 parent f7a02ad commit 5372f6a

File tree

24 files changed

+282
-153
lines changed

24 files changed

+282
-153
lines changed

docs/book/src/cronjob-tutorial/testdata/project/cmd/main.go

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -123,16 +123,17 @@ func main() {
123123
metricsServerOptions := metricsserver.Options{
124124
BindAddress: metricsAddr,
125125
SecureServing: secureMetrics,
126-
// TODO(user): TLSOpts is used to allow configuring the TLS config used for the server. If certificates are
127-
// not provided, self-signed certificates will be generated by default. This option is not recommended for
128-
// production environments as self-signed certificates do not offer the same level of trust and security
129-
// as certificates issued by a trusted Certificate Authority (CA). The primary risk is potentially allowing
130-
// unauthorized access to sensitive metrics data. Consider replacing with CertDir, CertName, and KeyName
131-
// to provide certificates, ensuring the server communicates using trusted and secure certificates.
132-
TLSOpts: tlsOpts,
126+
TLSOpts: tlsOpts,
133127
}
134128

135129
if secureMetrics {
130+
// TODO(user): If cert-manager is enabled under config/default/kustomizaton.yaml, you can uncomment the following
131+
// lines to use the certificate managed by cert-manager, mounted as a Kubernetes secret named 'serving-cert'.
132+
// This setup is recommended for production environments to ensure trusted and secure communication.
133+
// metricsServerOptions.CertDir = "/var/run/secrets/kubernetes.io/certs"
134+
// metricsServerOptions.CertName = "tls.crt"
135+
// metricsServerOptions.KeyName = "tls.key"
136+
136137
// FilterProvider is used to protect the metrics endpoint with authn/authz.
137138
// These configurations ensure that only authorized users and service accounts
138139
// can access the metrics endpoint. The RBAC are configured in 'config/rbac/kustomization.yaml'. More info:

docs/book/src/cronjob-tutorial/testdata/project/config/default/kustomization.yaml

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,8 +41,15 @@ patches:
4141
target:
4242
kind: Deployment
4343

44+
# [CERTMANAGER] The following patch configures the ServiceMonitor under ../prometheus
45+
# to securely reference certificates created and managed by cert-manager. Also, uncomment the
46+
# [WEBHOOK] patch bellow to mount the "serving-cert" in the Manager Deployment.
47+
#- path: ../prometheus/monitor_tls_patch.yaml
48+
#target:
49+
# kind: ServiceMonitor
50+
4451
# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in
45-
# crd/kustomization.yaml
52+
# crd/kustomization.yaml.
4653
- path: manager_webhook_patch.yaml
4754

4855
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER' prefix.

docs/book/src/cronjob-tutorial/testdata/project/config/prometheus/monitor.yaml

Lines changed: 4 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -16,14 +16,10 @@ spec:
1616
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
1717
tlsConfig:
1818
# TODO(user): The option insecureSkipVerify: true is not recommended for production since it disables
19-
# certificate verification. This poses a significant security risk by making the system vulnerable to
20-
# man-in-the-middle attacks, where an attacker could intercept and manipulate the communication between
21-
# Prometheus and the monitored services. This could lead to unauthorized access to sensitive metrics data,
22-
# compromising the integrity and confidentiality of the information.
23-
# Please use the following options for secure configurations:
24-
# caFile: /etc/metrics-certs/ca.crt
25-
# certFile: /etc/metrics-certs/tls.crt
26-
# keyFile: /etc/metrics-certs/tls.key
19+
# certificate verification, exposing the system to potential man-in-the-middle attacks.
20+
# For production environments, it is recommended to use cert-manager for automatic TLS certificate management.
21+
# To apply this configuration, enable cert-manager and use the patch located at config/prometheus/servicemonitor_tls_patch.yaml,
22+
# which securely references the certificate from the 'serving-cert' secret.
2723
insecureSkipVerify: true
2824
selector:
2925
matchLabels:

docs/book/src/getting-started/testdata/project/cmd/main.go

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -103,16 +103,17 @@ func main() {
103103
metricsServerOptions := metricsserver.Options{
104104
BindAddress: metricsAddr,
105105
SecureServing: secureMetrics,
106-
// TODO(user): TLSOpts is used to allow configuring the TLS config used for the server. If certificates are
107-
// not provided, self-signed certificates will be generated by default. This option is not recommended for
108-
// production environments as self-signed certificates do not offer the same level of trust and security
109-
// as certificates issued by a trusted Certificate Authority (CA). The primary risk is potentially allowing
110-
// unauthorized access to sensitive metrics data. Consider replacing with CertDir, CertName, and KeyName
111-
// to provide certificates, ensuring the server communicates using trusted and secure certificates.
112-
TLSOpts: tlsOpts,
106+
TLSOpts: tlsOpts,
113107
}
114108

115109
if secureMetrics {
110+
// TODO(user): If cert-manager is enabled under config/default/kustomizaton.yaml, you can uncomment the following
111+
// lines to use the certificate managed by cert-manager, mounted as a Kubernetes secret named 'serving-cert'.
112+
// This setup is recommended for production environments to ensure trusted and secure communication.
113+
// metricsServerOptions.CertDir = "/var/run/secrets/kubernetes.io/certs"
114+
// metricsServerOptions.CertName = "tls.crt"
115+
// metricsServerOptions.KeyName = "tls.key"
116+
116117
// FilterProvider is used to protect the metrics endpoint with authn/authz.
117118
// These configurations ensure that only authorized users and service accounts
118119
// can access the metrics endpoint. The RBAC are configured in 'config/rbac/kustomization.yaml'. More info:

docs/book/src/getting-started/testdata/project/config/default/kustomization.yaml

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,8 +41,15 @@ patches:
4141
target:
4242
kind: Deployment
4343

44+
# [CERTMANAGER] The following patch configures the ServiceMonitor under ../prometheus
45+
# to securely reference certificates created and managed by cert-manager. Also, uncomment the
46+
# [WEBHOOK] patch bellow to mount the "serving-cert" in the Manager Deployment.
47+
#- path: ../prometheus/monitor_tls_patch.yaml
48+
#target:
49+
# kind: ServiceMonitor
50+
4451
# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in
45-
# crd/kustomization.yaml
52+
# crd/kustomization.yaml.
4653
#- path: manager_webhook_patch.yaml
4754

4855
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER' prefix.

docs/book/src/getting-started/testdata/project/config/prometheus/monitor.yaml

Lines changed: 4 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -16,14 +16,10 @@ spec:
1616
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
1717
tlsConfig:
1818
# TODO(user): The option insecureSkipVerify: true is not recommended for production since it disables
19-
# certificate verification. This poses a significant security risk by making the system vulnerable to
20-
# man-in-the-middle attacks, where an attacker could intercept and manipulate the communication between
21-
# Prometheus and the monitored services. This could lead to unauthorized access to sensitive metrics data,
22-
# compromising the integrity and confidentiality of the information.
23-
# Please use the following options for secure configurations:
24-
# caFile: /etc/metrics-certs/ca.crt
25-
# certFile: /etc/metrics-certs/tls.crt
26-
# keyFile: /etc/metrics-certs/tls.key
19+
# certificate verification, exposing the system to potential man-in-the-middle attacks.
20+
# For production environments, it is recommended to use cert-manager for automatic TLS certificate management.
21+
# To apply this configuration, enable cert-manager and use the patch located at config/prometheus/servicemonitor_tls_patch.yaml,
22+
# which securely references the certificate from the 'serving-cert' secret.
2723
insecureSkipVerify: true
2824
selector:
2925
matchLabels:

docs/book/src/multiversion-tutorial/testdata/project/cmd/main.go

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -122,16 +122,17 @@ func main() {
122122
metricsServerOptions := metricsserver.Options{
123123
BindAddress: metricsAddr,
124124
SecureServing: secureMetrics,
125-
// TODO(user): TLSOpts is used to allow configuring the TLS config used for the server. If certificates are
126-
// not provided, self-signed certificates will be generated by default. This option is not recommended for
127-
// production environments as self-signed certificates do not offer the same level of trust and security
128-
// as certificates issued by a trusted Certificate Authority (CA). The primary risk is potentially allowing
129-
// unauthorized access to sensitive metrics data. Consider replacing with CertDir, CertName, and KeyName
130-
// to provide certificates, ensuring the server communicates using trusted and secure certificates.
131-
TLSOpts: tlsOpts,
125+
TLSOpts: tlsOpts,
132126
}
133127

134128
if secureMetrics {
129+
// TODO(user): If cert-manager is enabled under config/default/kustomizaton.yaml, you can uncomment the following
130+
// lines to use the certificate managed by cert-manager, mounted as a Kubernetes secret named 'serving-cert'.
131+
// This setup is recommended for production environments to ensure trusted and secure communication.
132+
// metricsServerOptions.CertDir = "/var/run/secrets/kubernetes.io/certs"
133+
// metricsServerOptions.CertName = "tls.crt"
134+
// metricsServerOptions.KeyName = "tls.key"
135+
135136
// FilterProvider is used to protect the metrics endpoint with authn/authz.
136137
// These configurations ensure that only authorized users and service accounts
137138
// can access the metrics endpoint. The RBAC are configured in 'config/rbac/kustomization.yaml'. More info:

docs/book/src/multiversion-tutorial/testdata/project/config/default/kustomization.yaml

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,8 +41,15 @@ patches:
4141
target:
4242
kind: Deployment
4343

44+
# [CERTMANAGER] The following patch configures the ServiceMonitor under ../prometheus
45+
# to securely reference certificates created and managed by cert-manager. Also, uncomment the
46+
# [WEBHOOK] patch bellow to mount the "serving-cert" in the Manager Deployment.
47+
#- path: ../prometheus/monitor_tls_patch.yaml
48+
#target:
49+
# kind: ServiceMonitor
50+
4451
# [WEBHOOK] To enable webhook, uncomment all the sections with [WEBHOOK] prefix including the one in
45-
# crd/kustomization.yaml
52+
# crd/kustomization.yaml.
4653
- path: manager_webhook_patch.yaml
4754

4855
# [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER' prefix.

docs/book/src/multiversion-tutorial/testdata/project/config/prometheus/monitor.yaml

Lines changed: 4 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -16,14 +16,10 @@ spec:
1616
bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
1717
tlsConfig:
1818
# TODO(user): The option insecureSkipVerify: true is not recommended for production since it disables
19-
# certificate verification. This poses a significant security risk by making the system vulnerable to
20-
# man-in-the-middle attacks, where an attacker could intercept and manipulate the communication between
21-
# Prometheus and the monitored services. This could lead to unauthorized access to sensitive metrics data,
22-
# compromising the integrity and confidentiality of the information.
23-
# Please use the following options for secure configurations:
24-
# caFile: /etc/metrics-certs/ca.crt
25-
# certFile: /etc/metrics-certs/tls.crt
26-
# keyFile: /etc/metrics-certs/tls.key
19+
# certificate verification, exposing the system to potential man-in-the-middle attacks.
20+
# For production environments, it is recommended to use cert-manager for automatic TLS certificate management.
21+
# To apply this configuration, enable cert-manager and use the patch located at config/prometheus/servicemonitor_tls_patch.yaml,
22+
# which securely references the certificate from the 'serving-cert' secret.
2723
insecureSkipVerify: true
2824
selector:
2925
matchLabels:

docs/book/src/reference/metrics.md

Lines changed: 54 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -136,40 +136,16 @@ spec:
136136
<aside class="warning">
137137
<h1>Changes Recommended for Production</h1>
138138

139-
The default scaffold to configure the metrics server in `cmd/main.go` uses `TLSOpts` that rely on self-signed certificates
140-
(SelfCerts), which are generated automatically. However, self-signed certificates are **not** recommended for production
141-
environments as they do not offer the same level of trust and security as certificates issued by a trusted
142-
Certificate Authority (CA).
139+
The default scaffold in `cmd/main.go` uses a **controller-runtime feature**
140+
to generate a certificate for securing the metrics server. While convenient
141+
for development and testing, this setup is **not recommended for production**.
143142

144-
While self-signed certificates are convenient for development and testing, they are unsuitable for production
145-
because they do not establish a chain of trust, making them vulnerable to security threats.
143+
Additionally, review the configuration file at `config/prometheus/monitor.yaml`
144+
to ensure secure integration with Prometheus. If `insecureSkipVerify: true` is
145+
enabled, certificate verification is turned off. **This is not recommended for production**
146+
as it exposes the system to man-in-the-middle attacks, potentially allowing
147+
unauthorized access to metrics data.
146148

147-
Furthermore, check the configuration file located at `config/prometheus/monitor.yaml` to
148-
ensure secure integration with Prometheus. If the `insecureSkipVerify: true` option is enabled,
149-
it means that certificate verification is turned off. This is **not** recommended for production as
150-
it poses a significant security risk by making the system vulnerable to man-in-the-middle attacks,
151-
where an attacker could intercept and manipulate the communication between Prometheus and the monitored services.
152-
This could lead to unauthorized access to metrics data, compromising the integrity and confidentiality of the information.
153-
154-
**In both cases, the primary risk is potentially allowing unauthorized access to sensitive metrics data.**
155-
156-
### Recommended Actions for a Secure Production Setup
157-
158-
1. **Replace Self-Signed Certificates:**
159-
- Instead of using `TLSOpts`, configure the `CertDir`, `CertName`, and `KeyName` options to use your own certificates.
160-
This ensures that your server communicates using trusted and secure certificates.
161-
162-
2. **Configure Prometheus Monitoring Securely:**
163-
- Check and update your Prometheus configuration file (`config/prometheus/monitor.yaml`) to ensure secure settings.
164-
- Replace `insecureSkipVerify: true` with the following secure options:
165-
166-
```yaml
167-
caFile: The path to the CA certificate file, e.g., /etc/metrics-certs/ca.crt.
168-
certFile: The path to the client certificate file, e.g., /etc/metrics-certs/tls.crt.
169-
keyFile: The path to the client key file, e.g., /etc/metrics-certs/tls.key.
170-
```
171-
172-
These settings ensure encrypted and authenticated communication between Prometheus and the monitored services, providing a secure monitoring setup.
173149
</aside>
174150

175151
<aside class="note">
@@ -187,6 +163,52 @@ An [issue](https://github.com/kubernetes-sigs/controller-runtime/issues/2781) ha
187163
enhance the controller-runtime and address these considerations.
188164
</aside>
189165

166+
### By exposing the metrics endpoint using HTTPS and Cert-Manager
167+
168+
Integrating `cert-manager` with your metrics service enables secure
169+
HTTPS access via TLS encryption. Follow the steps below to configure
170+
your project to expose the metrics endpoint using HTTPS with cert-manager.
171+
172+
1. **Enable Cert-Manager in `config/default/kustomization.yaml`:**
173+
- Uncomment the cert-manager resource to include it in your project:
174+
175+
```yaml
176+
- ../certmanager
177+
```
178+
179+
2. **Enable the Patch for the `ServiceMonitor` to Use the Cert-Manager-Managed Secret:**
180+
- Add or uncomment the `ServiceMonitor` patch to securely reference the cert-manager-managed secret, replacing insecure configurations with secure certificate verification:
181+
182+
```yaml
183+
- path: ../prometheus/monitor_tls_patch.yaml
184+
target:
185+
kind: ServiceMonitor
186+
```
187+
188+
3. **Enable the Patch to Mount the Cert-Manager-Managed Secret in the Controller Deployment:**
189+
- Use the `manager_webhook_patch.yaml` (or create a custom metrics patch) to mount the `serving-cert` secret in the Manager Deployment. This makes the cert-manager-managed certificate available for the metrics endpoint without enabling webhooks:
190+
191+
```yaml
192+
- path: manager_webhook_patch.yaml
193+
```
194+
195+
4. **Update `cmd/main.go` to Use the Certificate Managed by Cert-Manager:**
196+
- Modify `cmd/main.go` to configure the metrics server to use the cert-manager-managed certificates.
197+
Uncomment the lines for `CertDir`, `CertName`, and `KeyName`:
198+
199+
```go
200+
if secureMetrics {
201+
// TODO(user): If cert-manager is enabled under config/default/kustomization.yaml, you can uncomment the following
202+
// lines to use the certificate managed by cert-manager, mounted as a Kubernetes secret named 'serving-cert'.
203+
// This setup is recommended for production environments to ensure trusted and secure communication.
204+
// metricsServerOptions.CertDir = "/var/run/secrets/kubernetes.io/certs"
205+
// metricsServerOptions.CertName = "tls.crt"
206+
// metricsServerOptions.KeyName = "tls.key"
207+
}
208+
```
209+
210+
By following these steps, you’ll configure your metrics endpoint to securely expose metrics over HTTPS using cert-manager-managed
211+
certificates, improving security for production environments.
190212

191213
### By using Network Policy (You can optionally enable)
192214

@@ -202,16 +224,6 @@ Uncomment the following line in the `config/default/kustomization.yaml`:
202224
#- ../network-policy
203225
```
204226

205-
### By exposing the metrics endpoint using HTTPS and CertManager
206-
207-
Integrating `cert-manager` with your metrics service can secure the endpoint via TLS encryption.
208-
209-
To modify your project setup to expose metrics using HTTPS with
210-
the help of cert-manager, you'll need to change the configuration of both
211-
the `Service` under `config/default/metrics_service.yaml` and
212-
the `ServiceMonitor` under `config/prometheus/monitor.yaml` to use a secure HTTPS port
213-
and ensure the necessary certificate is applied.
214-
215227
## Exporting Metrics for Prometheus
216228

217229
Follow the steps below to export the metrics using the Prometheus Operator:

0 commit comments

Comments
 (0)