Skip to content

KFLUXINFRA-2558 - Add network policies to ESO (staging only) #9370

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+263 −0
Open

KFLUXINFRA-2558 - Add network policies to ESO (staging only) #9370

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions components/external-secrets-operator/staging/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,6 @@ apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base
- networkpolicy-default-deny.yaml
- networkpolicy-allow-egress.yaml
- networkpolicy-allow-ingress.yaml
135 changes: 135 additions & 0 deletions components/external-secrets-operator/staging/networkpolicy-allow-egress.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
---
# Allow egress to Kubernetes API server and DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-egress-apiserver-dns
namespace: external-secrets-operator
spec:
podSelector:
matchLabels:
app.kubernetes.io/instance: external-secrets
app.kubernetes.io/name: external-secrets
policyTypes:
- Egress
egress:
# Allow DNS resolution
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: openshift-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# Allow access to Kubernetes API server
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: openshift-kube-apiserver
ports:
- protocol: TCP
port: 6443
# Allow communication to the local webhook service
- to:
- podSelector:
matchLabels:
external-secrets.io/component: webhook # Selects the Webhook pod
ports:
- protocol: TCP
port: 9443 # Default webhook port
# Allow access to external secret providers (internet egress)
# This is needed for cloud providers like AWS Secrets Manager, Vault, etc.
- to:
- namespaceSelector: {}
ports:
- protocol: TCP
port: 443
- protocol: TCP
port: 8200 # Vault default port
- ipBlock:
Copy link
Contributor

@hugares hugares Nov 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you explain what is this CIDR and will this be the same for all clusters

Copy link
Contributor Author

@oswcab oswcab Nov 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This CIDR contains the IP for the kubernetes default ClusterAPI service, kubernetes.default.svc.cluster.local, which IP is 172.30.0.1 by default, and other kubernetes pods. If no access is granted to this IP, we receive this error:

025-11-26T20:19:29.323452-0500 - "- -- -" - - failed to inject conversion webhook 
CustomResourceDefinition: {"name":"clustersecretstores.external-secrets.io"}                       
 error: Get "https://172.30.0.1:443/api/v1/namespaces/external-secrets/secrets/external-secrets-webhook": dial tcp 172.30.0.1

Now, we can consider restricting this to the IP of the default k8s service, 172.30.0.1/32, but we'd need to make sure no other issues arise from this.

Copy link
Contributor Author

@oswcab oswcab Nov 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/hold

Just discovered another rule we need to make the health check work consistently

# This CIDR range covers the default Cluster IP (172.30.0.1) in OpenShift/K8s environments.
cidr: 172.30.0.0/16
ports:
- protocol: TCP
port: 443
---
# Allow webhook egress for cert management
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-webhook-egress
namespace: external-secrets-operator
spec:
podSelector:
matchLabels:
app.kubernetes.io/instance: external-secrets
app.kubernetes.io/name: external-secrets-webhook
policyTypes:
- Egress
egress:
# Allow DNS
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: openshift-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# Allow API server access
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: openshift-kube-apiserver
ports:
- protocol: TCP
port: 6443
- ipBlock:
# This CIDR range covers the default Cluster IP (172.30.0.1) in OpenShift/K8s environments.
cidr: 172.30.0.0/16
ports:
- protocol: TCP
port: 443
---
# Allow cert-controller egress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-cert-controller-egress
namespace: external-secrets-operator
spec:
podSelector:
matchLabels:
app.kubernetes.io/instance: external-secrets
app.kubernetes.io/name: external-secrets-cert-controller
policyTypes:
- Egress
egress:
# Allow DNS
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: openshift-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# Allow API server access
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: openshift-kube-apiserver
ports:
- protocol: TCP
port: 6443
- ipBlock:
# This CIDR range covers the default Cluster IP (172.30.0.1) in OpenShift/K8s environments.
cidr: 172.30.0.0/16
ports:
- protocol: TCP
port: 443

112 changes: 112 additions & 0 deletions components/external-secrets-operator/staging/networkpolicy-allow-ingress.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
---
# Allow ingress for validating webhook from Kubernetes API server
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-webhook-ingress
namespace: external-secrets-operator
spec:
podSelector:
matchLabels:
app.kubernetes.io/instance: external-secrets
app.kubernetes.io/name: external-secrets-webhook
policyTypes:
- Ingress
ingress:
# Allow webhook traffic from API server (required for CRD validation)
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: openshift-kube-apiserver
ports:
- protocol: TCP
port: 10250 # Webhook port
# Allow internal communication from the main ESO controller (required for validation checks)
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: external-secrets # Selector for the main controller
ports:
- protocol: TCP
port: 10250
---
# Allow metrics scraping from monitoring namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-metrics-ingress
namespace: external-secrets-operator
spec:
# Selects ALL ESO Pods (Controller, Webhook, Cert Controller)
podSelector:
matchLabels:
app.kubernetes.io/instance: external-secrets
policyTypes:
- Ingress
ingress:
# Allow Prometheus to scrape metrics
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: appstudio-workload-monitoring
ports:
- protocol: TCP
port: 8080 # Metrics port
# Also allow from openshift-monitoring if it exists
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: openshift-monitoring
ports:
- protocol: TCP
port: 8080
---
# Allow health checks from kubelet (for liveness/readiness probes)
# Health checks come from the node network, so we allow from all namespaces
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-health-checks
namespace: external-secrets-operator
spec:
# Selects ALL ESO Pods (Controller, Webhook, Cert Controller)
podSelector:
matchLabels:
app.kubernetes.io/instance: external-secrets
policyTypes:
- Ingress
ingress:
# Allow health checks on port 8081 (used by cert-controller and webhook)
# Allow from all namespaces (kubelet health checks come from node network)
# This is safe because health checks are read-only and internal to the cluster
- from:
- namespaceSelector: {} # Allow from all namespaces (includes node network)
ports:
- protocol: TCP
port: 8081 # Health check port for readiness/liveness probes

---
# Allow intra-namespace communication
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-same-namespace
namespace: external-secrets-operator
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: external-secrets
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
app.kubernetes.io/name: external-secrets
egress:
- to:
- podSelector:
matchLabels:
app.kubernetes.io/name: external-secrets

13 changes: 13 additions & 0 deletions components/external-secrets-operator/staging/networkpolicy-default-deny.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
# Default deny all traffic in external-secrets-operator namespace
# This provides baseline security - all other NetworkPolicies will whitelist specific traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: external-secrets-operator
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
Loading