Skip to content

OCPBUGS-9037, OCPBUGS-64565: Use cluster wildcard certificate for ingress canary #1155

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

OCPBUGS-9037, OCPBUGS-64565: Use cluster wildcard certificate for ingress canary #1155

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pkg/manifests/assets/canary/daemonset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ spec:
volumes:
- name: cert
secret:
secretName: canary-serving-cert
# secret name is set at runtime
defaultMode: 0420
updateStrategy:
type: RollingUpdate
Expand Down
4 changes: 1 addition & 3 deletions pkg/manifests/assets/canary/service.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,8 @@
# Specific values are applied at runtime
kind: Service
apiVersion: v1
metadata:
metadata: {}
# name and namespace are set at runtime.
Copy link
Contributor

@candita candita Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
# name and namespace are set at runtime.
# metadata values are set at runtime.

annotations:
service.beta.openshift.io/serving-cert-secret-name: canary-serving-cert
spec:
type: ClusterIP
ports:
Expand Down
24 changes: 22 additions & 2 deletions pkg/operator/controller/canary/daemonset.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,23 @@ import (
"github.com/google/go-cmp/cmp"
"github.com/google/go-cmp/cmp/cmpopts"

operatorv1 "github.com/openshift/api/operator/v1"
"github.com/openshift/cluster-ingress-operator/pkg/manifests"
"github.com/openshift/cluster-ingress-operator/pkg/operator/controller"

appsv1 "k8s.io/api/apps/v1"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/types"
)

// ensureCanaryDaemonSet ensures the canary daemonset exists
func (r *reconciler) ensureCanaryDaemonSet() (bool, *appsv1.DaemonSet, error) {
desired := desiredCanaryDaemonSet(r.config.CanaryImage)
secretName, err := r.canarySecretName(controller.CanaryDaemonSetName().Namespace)
if err != nil {
return false, nil, err
}
Comment on lines +22 to +25
Copy link
Contributor

@candita candita Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not get the secretName within desiredCanaryDaemonSet? Then you don't need to pass it in as a parameter.

Copy link
Contributor

@Miciah Miciah Nov 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In order to enable desiredCanaryDaemonSet to get the secret, we would need to pass the client to desiredCanaryDaemonSet. Passing the client to desiredCanaryDaemonSet would go against our established pattern that ensureFoo gets the necessary inputs for desiredFoo and passes those specific inputs to desiredFoo. Passing the client would also mean we would be adding a parameter to desiredCanaryDaemonSet anyway, so it wouldn't really have any benefit over the current change.

desired := desiredCanaryDaemonSet(r.config.CanaryImage, secretName.Name)
haveDs, current, err := r.currentCanaryDaemonSet()
if err != nil {
return false, nil, err
Expand Down Expand Up @@ -80,7 +86,7 @@ func (r *reconciler) updateCanaryDaemonSet(current, desired *appsv1.DaemonSet) (

// desiredCanaryDaemonSet returns the desired canary daemonset read in
// from manifests
func desiredCanaryDaemonSet(canaryImage string) *appsv1.DaemonSet {
func desiredCanaryDaemonSet(canaryImage, secretName string) *appsv1.DaemonSet {
daemonset := manifests.CanaryDaemonSet()
name := controller.CanaryDaemonSetName()
daemonset.Name = name.Name
Expand All @@ -97,6 +103,8 @@ func desiredCanaryDaemonSet(canaryImage string) *appsv1.DaemonSet {
daemonset.Spec.Template.Spec.Containers[0].Image = canaryImage
daemonset.Spec.Template.Spec.Containers[0].Command = []string{"ingress-operator", CanaryHealthcheckCommand}

daemonset.Spec.Template.Spec.Volumes[0].Secret.SecretName = secretName
Copy link
Contributor

@candita candita Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be better to call canarySecretName right here instead of in the ensure function, and then passing it in here.

Suggested change
daemonset.Spec.Template.Spec.Volumes[0].Secret.SecretName = secretName
daemonset.Spec.Template.Spec.Volumes[0].Secret.SecretName, err := canarySecretName(name.Namespace)

Copy link
Contributor

@candita candita Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And then you could test the secret generation in Test_desiredCanaryDaemonSet instead of writing a new test for ensureCanaryDaemonSet.


return daemonset
}

Expand Down Expand Up @@ -196,3 +204,15 @@ func cmpTolerations(a, b corev1.Toleration) bool {
}
return true
}

func (r *reconciler) canarySecretName(Namespace string) (types.NamespacedName, error) {
defaultIC := operatorv1.IngressController{}
defaultICName := types.NamespacedName{
Name: manifests.DefaultIngressControllerName,
Namespace: r.config.Namespace,
}
if err := r.client.Get(context.TODO(), defaultICName, &defaultIC); err != nil {
return types.NamespacedName{}, err
}
return controller.RouterEffectiveDefaultCertificateSecretName(&defaultIC, Namespace), nil
}
22 changes: 20 additions & 2 deletions pkg/operator/controller/canary/daemonset_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ import (
func Test_desiredCanaryDaemonSet(t *testing.T) {
// canaryImageName is the ingress-operator image
canaryImageName := "openshift/origin-cluster-ingress-operator:latest"
daemonset := desiredCanaryDaemonSet(canaryImageName)
certSecretName := "test_secret_name"
daemonset := desiredCanaryDaemonSet(canaryImageName, certSecretName)

expectedDaemonSetName := controller.CanaryDaemonSetName()

Expand Down Expand Up @@ -83,6 +84,23 @@ func Test_desiredCanaryDaemonSet(t *testing.T) {
if !cmp.Equal(tolerations, expectedTolerations) {
t.Errorf("expected daemonset tolerations to be %v, but got %v", expectedTolerations, tolerations)
}

volumes := daemonset.Spec.Template.Spec.Volumes
secretMode := int32(0420)
expectedVolumes := []corev1.Volume{
{
Name: "cert",
VolumeSource: corev1.VolumeSource{
Secret: &corev1.SecretVolumeSource{
SecretName: certSecretName,
DefaultMode: &secretMode,
},
},
},
}
if !cmp.Equal(volumes, expectedVolumes) {
t.Errorf("expected daemonset volumes to be %v, but got %v", expectedVolumes, volumes)
}
}

func Test_canaryDaemonsetChanged(t *testing.T) {
Expand Down Expand Up @@ -229,7 +247,7 @@ func Test_canaryDaemonsetChanged(t *testing.T) {

for _, tc := range testCases {
t.Run(tc.description, func(t *testing.T) {
original := desiredCanaryDaemonSet("")
original := desiredCanaryDaemonSet("", "foobar")
mutated := original.DeepCopy()
tc.mutate(mutated)
if changed, updated := canaryDaemonSetChanged(original, mutated); changed != tc.expect {
Expand Down
10 changes: 6 additions & 4 deletions pkg/operator/controller/canary/service_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import (
"testing"

"github.com/google/go-cmp/cmp"
"github.com/google/go-cmp/cmp/cmpopts"

"github.com/openshift/cluster-ingress-operator/pkg/manifests"
"github.com/openshift/cluster-ingress-operator/pkg/operator/controller"
Expand Down Expand Up @@ -54,10 +55,8 @@ func Test_desiredCanaryService(t *testing.T) {
t.Errorf("expected service owner references %#v, but got %#v", expectedOwnerRefs, service.OwnerReferences)
}

expectedAnnotations := map[string]string{
"service.beta.openshift.io/serving-cert-secret-name": "canary-serving-cert",
}
if !cmp.Equal(service.Annotations, expectedAnnotations) {
expectedAnnotations := map[string]string{}
if !cmp.Equal(service.Annotations, expectedAnnotations, cmpopts.EquateEmpty()) {
t.Errorf("expected service annotations to be %q, but got %q", expectedAnnotations, service.Annotations)
}

Expand Down Expand Up @@ -90,6 +89,9 @@ func Test_canaryServiceChanged(t *testing.T) {
{
description: "changed annotation",
mutate: func(service *corev1.Service) {
if service.Annotations == nil {
service.Annotations = map[string]string{}
}
service.Annotations["foo"] = "bar"
},
expected: true,
Expand Down
28 changes: 28 additions & 0 deletions pkg/operator/controller/certificate/controller.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import (
"time"

logf "github.com/openshift/cluster-ingress-operator/pkg/log"
"github.com/openshift/cluster-ingress-operator/pkg/manifests"
"github.com/openshift/cluster-ingress-operator/pkg/operator/controller"
ingresscontroller "github.com/openshift/cluster-ingress-operator/pkg/operator/controller/ingress"

Expand Down Expand Up @@ -105,6 +106,33 @@ func (r *reconciler) Reconcile(ctx context.Context, request reconcile.Request) (
if _, err := r.ensureDefaultCertificateForIngress(ca, deployment.Namespace, deploymentRef, ingress); err != nil {
errs = append(errs, fmt.Errorf("failed to ensure default cert for %s: %v", ingress.Name, err))
}
if ingress.Name == manifests.DefaultIngressControllerName {
Copy link
Contributor

@Miciah Miciah Nov 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a comment to explain why this logic only applies to the "default" IngressController. Namely, the canary check is specifically for verifying that the default IngressController is ready.

log.Info("ensuring canary certificate")
Copy link
Contributor

@Miciah Miciah Nov 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
log.Info("ensuring canary certificate")
log.Info("Ensuring canary certificate")

See https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md#message-style-guidelines.

daemonset := &appsv1.DaemonSet{}
err = r.client.Get(ctx, controller.CanaryDaemonSetName(), daemonset)
if err != nil {
if errors.IsNotFound(err) {
// All ingresses should have a deployment, so this one may not have been
// created yet. Retry after a reasonable amount of time.
log.Info("canary daemonset not found; will retry default cert sync")
result.RequeueAfter = 5 * time.Second
} else {
errs = append(errs, fmt.Errorf("failed to get daemonset: %v", err))
Copy link
Contributor

@Miciah Miciah Nov 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use %w when wrapping errors, on line 120 and on line 132.

}
} else {
trueVar := true
canaryRef := metav1.OwnerReference{
APIVersion: "apps/v1",
Kind: "Daemonset",
Name: daemonset.Name,
UID: daemonset.UID,
Controller: &trueVar,
}
if _, err := r.ensureDefaultCertificateForIngress(ca, "openshift-ingress-canary", canaryRef, ingress); err != nil {
Copy link
Contributor

@candita candita Nov 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't there need to be a different function to ensure a canary cert rather than ensure a default cert? Does this ensure the correct cert?

Copy link
Contributor

@Miciah Miciah Nov 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intent is that the canary application should use the default IngressController's default certificate. This way, as long as the default IngressController has a correctly configured default certificate, so too will the canary application. Because #978 changed the canary application to use a TLS passthrough route, the only way to have the canary application use the default IngressController's default certificate is to copy that certificate to the canary application's namespace and configure the application to use that copy of the certificate.

If I understand correctly, ensureDefaultCertificateForIngress actually generates a new server certificate, using the existing CA certificate, so this logic doesn't quite implement the intent. We could use r.ensureDefaultCertificateForIngress(ca, ingress.Namespace, ref, ingress) (note the namespace) to get the existing certificate (or create it if it's missing), but it seems simpler just to do a Get from ingress.Namespace and then Create in "openshift-ingress-canary".

errs = append(errs, fmt.Errorf("failed to ensure canary cert for %s: %v", ingress.Name, err))
}
}
}
}
}

Expand Down
4 changes: 2 additions & 2 deletions pkg/operator/controller/certificate/default_cert.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,14 +48,14 @@ func (r *reconciler) ensureDefaultCertificateForIngress(caSecret *corev1.Secret,
if deleted, err := r.deleteRouterDefaultCertificate(current); err != nil {
return true, fmt.Errorf("failed to delete default certificate: %v", err)
} else if deleted {
r.recorder.Eventf(ci, "Normal", "DeletedDefaultCertificate", "Deleted default wildcard certificate %q", current.Name)
r.recorder.Eventf(ci, "Normal", "DeletedDefaultCertificate", "Deleted default wildcard certificate %q in namespace %q", current.Name, current.Namespace)
return false, nil
}
case wantCert && !haveCert:
if created, err := r.createRouterDefaultCertificate(desired); err != nil {
return false, fmt.Errorf("failed to create default certificate: %v", err)
} else if created {
r.recorder.Eventf(ci, "Normal", "CreatedDefaultCertificate", "Created default wildcard certificate %q", desired.Name)
r.recorder.Eventf(ci, "Normal", "CreatedDefaultCertificate", "Created default wildcard certificate %q in namespace %q", desired.Name, desired.Namespace)
return true, nil
}
case wantCert && haveCert:
Expand Down