Skip to content
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,20 @@
"source": "openshift:payload:cluster-version-operator",
"lifecycle": "blocking",
"environmentSelector": {}
},
{
"name": "[Jira:Cluster Version Operator] The cluster version operator should not install resources annotated with release.openshift.io/delete=true",
"labels": {
"42543": {},
"Conformance": {},
"High": {},
"cvo": {}
},
"resources": {
"isolation": {}
},
"source": "openshift:payload:cluster-version-operator",
"lifecycle": "blocking",
"environmentSelector": {}
}
]
82 changes: 77 additions & 5 deletions test/cvo/cvo.go
Original file line number Diff line number Diff line change
@@ -1,12 +1,84 @@
package cvo

Copy link
Contributor

@DavidHurta DavidHurta Nov 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment not related to the code

I will hold us to a higher standard regarding the git hygiene, if I may.

Please see the Contributing guide for the cluster-version-operator repository.

I am mainly concerned about the following points:

  • Make commits of logical units.

  • Make sure your commit messages are in the proper format (see below).

  • The PR title.

Currently, the PR consists of a single commit jianl - First e2e test, which consists of multiple logical units, and the commit title does not follow the recommended format. While PR titles are not explicitly mentioned in the guide, a PR title should state what a PR distinctly and briefly does.

Please keep in mind that these guidelines have their practical purposes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the commit, because this is the first OTE PR, it contains many logic, so the description may not describes everything. I will keep PR simple and describe the logic in the future.

import (
. "github.com/onsi/ginkgo/v2"
. "github.com/onsi/gomega"
"context"
"fmt"

g "github.com/onsi/ginkgo/v2"
o "github.com/onsi/gomega"
kerrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"

v1 "github.com/openshift/client-go/config/clientset/versioned/typed/config/v1"
"github.com/openshift/cluster-version-operator/test/utilities"
)

var _ = Describe(`[Jira:"Cluster Version Operator"] cluster-version-operator-tests`, func() {
It("should support passing tests", func() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is my sanity test being removed? :(

Copy link
Contributor

@DavidHurta DavidHurta Nov 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its future usefulness is questionable. However, for the moment, I am using it to validate that the openshift/origin#30316 PR is functional. Would it be okay for me rename to the component in the [Jira:Cluster Version Operator] cluster-version-operator-tests should support passing tests test case in another PR to get it merged quickly, then merge the origin PR, and for this PR to just ignore the test case? The test case is not resource-intensive, and it provides a useful "if you can't see this test, then something is wrong with the integration" signal.

/cc @hongkailiu

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can then validate this PR with an integrated OTE in the CVO repository.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you want to keep "should support passing tests" only temporarily (to validate and merge openshift/origin#30316), and then do ext.IgnoreObsoleteTests() to ignore it with this pull?
😄

I would not practice ext.IgnoreObsoleteTests() this way as it creates codes it stays forever in our CVO repo.
I would avoid renaming, or deleting if i could.

Maybe it is not a big deal, and it is OK delete as long as having them ignored?
Honestly, I do not see that far. So if it makes your work much easier, I am fine with it too.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't mind just keeping the test forever.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and for this PR to just ignore the test case?

I meant "ignore" as is in, do not touch. Not as in utilizing the ext.IgnoreObsoleteTests functionality.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack.

Jian has reverted the deleting.
Let us keep it forever. 😃

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DavidHurta I just added back the case and rename it as you mentioned. It is a independent test suite.

Copy link
Contributor

@DavidHurta DavidHurta Nov 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. It helps with the integration. It also provides a signal of the integration working in the future. Such sanity tests are also in several other OpenShift repositories that already integrated the OTE or are in the process of the integration.

Expect(true).To(BeTrue())
var _ = g.Describe(`[Jira:"Cluster Version Operator"] cluster-version-operator-tests`, func() {
g.It("should support passing tests", func() {
o.Expect(true).To(o.BeTrue())
})
})

var _ = g.Describe("[Jira:Cluster Version Operator] The cluster version operator", g.Ordered, g.Label("cvo"), func() {
defer g.GinkgoRecover()
var client *v1.ConfigV1Client
var kubeclient *kubernetes.Clientset

g.BeforeAll(func() {
client = utilities.MustGetV1Client()
kubeclient = utilities.MustGetKubeClient()
})

g.It(`should not install resources annotated with release.openshift.io/delete=true`, g.Label("Conformance", "High", "42543"), func() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume that the 42543 label represents the polarion ID test case. Is that correct? If so, is it common or expected to write these IDs in labels?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is polarion ID, I have no idea if it is common but I think it is helpful for us when we want to read the case from polarion.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does the High label represent, please? Can you link me a reference/docs for the label?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High is a property of the case on polarion, with it we can create test suites with same level, then we can run test suite by case level.
For example, we want to run all High level test cases on a special version, then we can follow below steps:

	ext.AddSuite(extension.Suite{
		Name:       "high level test cases",
		Parents:    []string{"openshift/conformance"},
		Qualifiers: []string{`"High" in labels`},
	})

Then run it:

$ _output/linux/amd64/cluster-version-operator-tests run-suite "high level test cases"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"High" in labels means the High keyword in labels list, labels can be found by _output/linux/amd64/cluster-version-operator-tests list, for example:

$ _output/linux/amd64/cluster-version-operator-tests list
[
  {
    "name": "[Jira:\"Cluster Version Operator\"] cluster-version-operator-tests should support passing tests the sanity test should pass",
    "labels": {},
    "resources": {
      "isolation": {}
    },
    "source": "openshift:payload:cluster-version-operator",
    "lifecycle": "blocking",
    "environmentSelector": {}
  },
  {
    "name": "[Jira:\"Cluster Version Operator\"] The cluster version operator should not install resources annotated with release.openshift.io/delete=true",
    "labels": {
      "42543": {},
      "Conformance": {},
      "High": {},
      "cvo": {}
    },
    "resources": {
      "isolation": {}
    },
    "source": "openshift:payload:cluster-version-operator",
    "lifecycle": "blocking",
    "environmentSelector": {}
  }
]

Similarly, we can use name.contains("[Serial]") in Qualifiers.

annotation := "release.openshift.io/delete"

auths, err := client.Authentications().List(context.TODO(), metav1.ListOptions{})
o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing authentications")

g.By(fmt.Sprintf("checking if authentication with %s annotation exists", annotation))
for _, auth := range auths.Items {
if _, ok := auth.Annotations[annotation]; ok {
o.Expect(ok).NotTo(o.BeTrue(), fmt.Sprintf("Unexpectedly installed authentication %s which has '%s' annotation", auth.Name, annotation))
}
}

namespaces, err := kubeclient.CoreV1().Namespaces().List(context.TODO(), metav1.ListOptions{})
o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing namespaces")

g.By(fmt.Sprintf("checking if special resources with %s annotation exist in all namespaces", annotation))
for _, ns := range namespaces.Items {
namespace := ns.Name
fmt.Printf("namespace: %s\n", namespace)

fmt.Println(" - Test services...")
services, err := kubeclient.CoreV1().Services(namespace).List(context.TODO(), metav1.ListOptions{})
o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing services")
for _, service := range services.Items {
if _, ok := service.Annotations[annotation]; ok {
o.Expect(ok).NotTo(o.BeTrue(), fmt.Sprintf("Unexpectedly installed service %s which has '%s' annotation", service.Name, annotation))
}
}

fmt.Println(" - Test RoleBinding...")
rolebindings, err := kubeclient.RbacV1().RoleBindings(namespace).List(context.TODO(), metav1.ListOptions{})
o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing rolebindings")
for _, rb := range rolebindings.Items {
if _, ok := rb.Annotations[annotation]; ok {
o.Expect(ok).NotTo(o.BeTrue(), fmt.Sprintf("Unexpectedly installed RoleBinding %s which has '%s' annotation", rb.Name, annotation))
}
}

fmt.Println(" - Test CronJob...")
cronjobs, err := kubeclient.BatchV1().CronJobs(namespace).List(context.TODO(), metav1.ListOptions{})
o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing cronjobs")
for _, cj := range cronjobs.Items {
if _, ok := cj.Annotations[annotation]; ok {
o.Expect(ok).NotTo(o.BeTrue(), fmt.Sprintf("Unexpectedly installed CronJob %s which has %s annotation", cj.Name, annotation))
}
}

fmt.Println("success")
}
Comment on lines +91 to +150
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

Fix List error handling to fail on real API errors, not just NotFound

All the List calls currently do:

auths, err := client.Authentications().List(...)
o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing ...")

This has two problems:

  1. False positives on failures

    • Any non-NotFound error (e.g., auth, RBAC, networking, API server issues) makes kerrors.IsNotFound(err) return false, so the expectation passes.
    • The test then proceeds, typically iterating over an empty/zero-valued list, and can still pass even though the underlying List failed.
    • That means the test can report success while not actually verifying the cluster state.
  2. Misleading messages

    • The description says “The NotFound error should occur when listing …” while the matcher asserts BeFalse(), which is contradictory.

You want the test to fail on any error from those List calls. A more robust pattern is to assert on err directly, for each List:

-		auths, err := client.Authentications().List(context.TODO(), metav1.ListOptions{})
-		o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing authentications")
+		auths, err := client.Authentications().List(context.TODO(), metav1.ListOptions{})
+		o.Expect(err).NotTo(o.HaveOccurred(), "failed to list authentications")

-		namespaces, err := kubeclient.CoreV1().Namespaces().List(context.TODO(), metav1.ListOptions{})
-		o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing namespaces")
+		namespaces, err := kubeclient.CoreV1().Namespaces().List(context.TODO(), metav1.ListOptions{})
+		o.Expect(err).NotTo(o.HaveOccurred(), "failed to list namespaces")

-			services, err := kubeclient.CoreV1().Services(namespace).List(context.TODO(), metav1.ListOptions{})
-			o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing services")
+			services, err := kubeclient.CoreV1().Services(namespace).List(context.TODO(), metav1.ListOptions{})
+			o.Expect(err).NotTo(o.HaveOccurred(), "failed to list services")

-			rolebindings, err := kubeclient.RbacV1().RoleBindings(namespace).List(context.TODO(), metav1.ListOptions{})
-			o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing rolebindings")
+			rolebindings, err := kubeclient.RbacV1().RoleBindings(namespace).List(context.TODO(), metav1.ListOptions{})
+			o.Expect(err).NotTo(o.HaveOccurred(), "failed to list rolebindings")

-			cronjobs, err := kubeclient.BatchV1().CronJobs(namespace).List(context.TODO(), metav1.ListOptions{})
-			o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing cronjobs")
+			cronjobs, err := kubeclient.BatchV1().CronJobs(namespace).List(context.TODO(), metav1.ListOptions{})
+			o.Expect(err).NotTo(o.HaveOccurred(), "failed to list cronjobs")

This way, any failure to talk to the API (including NotFound on the resource type) will correctly fail the spec instead of silently passing.

If you want to double‑check semantics, please confirm in the Kubernetes/client-go docs (or local experiments) that List should normally return an empty list with err == nil when no items exist, and that kerrors.IsNotFound is intended for handling Get/Delete style operations rather than normal successful Lists.


🌐 Web query:

How does the Kubernetes client-go List API signal errors vs. an empty result, and when is k8s.io/apimachinery/pkg/api/errors.IsNotFound typically used?

💡 Result:

Short answer: List returns an empty list with err == nil (the returned List object's Items slice is just empty). IsNotFound is not used for a successful List — it's used when an operation asking for a specific named resource (e.g., Get or Delete by name) returns HTTP 404; k8s.io/apimachinery/pkg/api/errors.IsNotFound(err) detects that 404 StatusError. Use IsNotFound when you expect a single resource might not exist and you want to handle "resource missing" specially.

References: k8s error helpers (IsNotFound). [1] Client List/Get semantics (List returns list object; Reader/List returns error only on failure). [2] REST practice: collection endpoints return 200 + empty array when no items. [3]

Sources:
[1] k8s.io/apimachinery/pkg/api/errors (IsNotFound description).
[2] sigs.k8s.io/controller-runtime/pkg/client (Client.List / Reader semantics).
[3] RESTful API design discussion (return 200 with empty array for empty collections).


Replace error assertions with proper nil checks for all List calls

The web search confirms your analysis: Kubernetes List operations return an empty list with err == nil on success. The IsNotFound check is not intended for List operations—it detects HTTP 404 on named-resource operations (Get/Delete), not on collection endpoints.

This means the current code has a critical flaw:

o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing ...")

—will pass whenever IsNotFound(err) == false, which includes:

  • Normal success with no items ✓
  • Network/auth/RBAC failures ✗ (silent pass when should fail)

Your proposed fix is correct. Replace all five List error checks in test/cvo/cvo.go (lines 26–27, 41–42, 51–52, 60–61, 68–69) with:

-o.Expect(kerrors.IsNotFound(err)).To(o.BeFalse(), "The NotFound error should occur when listing ...")
+o.Expect(err).NotTo(o.HaveOccurred(), "failed to list ...")

This ensures any API failure is caught, not silently suppressed.

🤖 Prompt for AI Agents
In test/cvo/cvo.go around lines 23 to 82, the List() error checks use
kerrors.IsNotFound(err) which is inappropriate for collection endpoints and can
silently ignore real API errors; replace each usage at lines ~26–27, ~41–42,
~51–52, ~60–61, and ~68–69 with a direct nil-check assertion (i.e., assert err
is nil) so any List failure causes the test to fail; keep the existing
descriptive messages but change the assertion to ensure err == nil.

})
})
71 changes: 71 additions & 0 deletions test/utilities/connection.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
package utilities

import (
"errors"
"fmt"
"os"

"k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
"k8s.io/client-go/tools/clientcmd"

configclientv1 "github.com/openshift/client-go/config/clientset/versioned/typed/config/v1"
)

// getKubeConfig get KUBECONFIG file from environment variable
func getKubeConfig() (*rest.Config, error) {
configPath, present := os.LookupEnv("KUBECONFIG")
if !present {
return nil, errors.New("the environment variable KUBECONFIG must be set")
}
config, err := clientcmd.BuildConfigFromFlags("", configPath)
return config, err
}

// getKubeClient creates a kubernetes.Clientset instance.
func getKubeClient() (*kubernetes.Clientset, error) {
config, err := getKubeConfig()
if err != nil {
return nil, fmt.Errorf("unable to load build config: %w", err)
}
// Create the Clientset
clientset, err := kubernetes.NewForConfig(config)
if err != nil {
return nil, fmt.Errorf("unable to create a Kubernetes clientset: %w", err)
}

return clientset, nil
}

// getV1Client creates a configclientv1.ConfigV1Client instance.
func getV1Client() (*configclientv1.ConfigV1Client, error) {
config, err := getKubeConfig()
if err != nil {
return nil, fmt.Errorf("unable to load build config: %w", err)
}
// Create the Clientset
clientset, err := configclientv1.NewForConfig(config)
if err != nil {
return nil, fmt.Errorf("unable to create a configclientv1 clientset: %w", err)
}

return clientset, nil
}

// MustGetKubeClient creates a kubernetes.Clientset instance, or panics on failures.
func MustGetKubeClient() *kubernetes.Clientset {
clientset, err := getKubeClient()
if err != nil {
panic("unable to create a Kubernetes clientset: " + err.Error())
}
return clientset
}

// MustGetV1Client creates a configclientv1.ConfigV1Client instance, or panics on failures.
func MustGetV1Client() *configclientv1.ConfigV1Client {
clientset, err := getV1Client()
if err != nil {
panic("unable to create a configclientv1 clientset: " + err.Error())
}
return clientset
}