Skip to content

CMP-3800: Add e2e test (33431) for ComplianceCheckResult label queries#981

Merged
rhmdnd merged 9 commits intoComplianceAsCode:masterfrom
taimurhafeez:add-test-33431
Mar 3, 2026
Merged

CMP-3800: Add e2e test (33431) for ComplianceCheckResult label queries#981
rhmdnd merged 9 commits intoComplianceAsCode:masterfrom
taimurhafeez:add-test-33431

Conversation

@taimurhafeez
Copy link
Collaborator

@taimurhafeez taimurhafeez commented Nov 21, 2025

Verify that ComplianceCheckResult objects can be queried using labels:

  • compliance.openshift.io/check-severity
  • compliance.openshift.io/check-status
  • compliance.openshift.io/scan-name
  • compliance.openshift.io/suite

Changes

  • Updated TestScanProducesRemediations to TestScanProducesRemediationsAndLabels in tests/e2e/parallel/main_test.go
  • Add AssertCheckResultByLabel helper in tests/e2e/framework/common.go
  1. Run the test in make e2e-parallel E2E_GO_TEST_FLAGS="-v -run TestScanProducesRemediationsAndLabels"

Tested on OCP 4.21 and expected output:

=== RUN   TestScanProducesRemediationsAndLabels
=== PAUSE TestScanProducesRemediationsAndLabels
=== CONT  TestScanProducesRemediationsAndLabels
2026/03/02 11:53:26 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2026/03/02 11:53:31 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2026/03/02 11:53:36 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2026/03/02 11:53:40 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2026/03/02 11:53:46 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2026/03/02 11:53:51 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: AGGREGATING
2026/03/02 11:53:56 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: AGGREGATING
2026/03/02 11:54:01 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: AGGREGATING
2026/03/02 11:54:11 ComplianceScan ready (DONE)
2026/03/02 11:54:11 All scans in ComplianceSuite have finished (test-scan-produces-remediations-and-labels)
osdk-e2e-9666bf4d-7b32-4cac-9127-9741af0d0080 map[compliance.openshift.io/suite:test-scan-produces-remediations-and-labels]
--- PASS: TestScanProducesRemediationsAndLabels (52.03s)
PASS

@openshift-ci
Copy link

openshift-ci bot commented Nov 21, 2025

Hi @taimurhafeez. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@xiaojiey
Copy link
Collaborator

Hi @taimurhafeez, I don't think it is meaningful to create a dedicate serial test case to check the labels for the ccr. If you really want to cover the test point, I think it is better to adding label queries to 1 existing parallel tests than creating a dedicated serial test

// NodeSelector: workerNodesLabel,
// ComplianceScanSettings: compv1alpha1.ComplianceScanSettings{
// Debug: true,
// },
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would keep the curly bracket where it originally was, so that we do not merge changes that are not related to the test case

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just nitpicking with the prevous comment and here as well - the same applies to the commented lines in func createInvalidMachineConfigPool in ‎tests/e2e/framework/common.go and the comments in tests/e2e/framework/main_entry.go

I know it is early in the reviews, but I think it would be best to have it prepared exactly the way we would like to merge it :)

@Anna-Koudelkova
Copy link
Collaborator

The test case is PASSing succesfully on OCP 4.20 cluster. After running it, one of the worker nodes on my cluster has gone into degraded state though:

$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-2b8ffc374493b63e18f96d2aa594f092   True      False      False      3              3                   3                     0                      102m
worker   rendered-worker-033af400c2a339ce01799d80ce78313c   False     True       True       3              2                   2                     1                      102m

I am not really sure if this is just a coincidence or what is a rootcase yet, so far I just want to post it here so we have a trace that this has happened - in case it happens to someone else as well in the future

@taimurhafeez taimurhafeez changed the title Add e2e test (33431) for ComplianceCheckResult label queries CMP-3800: Add e2e test (33431) for ComplianceCheckResult label queries Dec 2, 2025
@openshift-ci-robot
Copy link
Collaborator

@taimurhafeez: This pull request references CMP-3800 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

Details

In response to this:

Verify that ComplianceCheckResult objects can be queried using labels:

  • compliance.openshift.io/check-severity
  • compliance.openshift.io/check-status
  • compliance.openshift.io/scan-name
  • compliance.openshift.io/suite

Changes

  • Updated TestScanProducesRemediations to TestScanProducesRemediationsAndLabels in tests/e2e/parallel/main_test.go
  • Add AssertCheckResultByLabel helper in tests/e2e/framework/common.go

-to implement this PR

  1. The following line needs to be commented in tests/e2e/framework/main_entry.go or there will be error error creating Machine Config Pool e2e-invalid.
    line to be commented --> return fmt.Errorf("failed to create Machine Config Pool %s: %w", "e2e-invalid", err)
  2. Run the test in make e2e-serial E2E_GO_TEST_FLAGS="-v -run TestScanProducesRemediationsAndLabels"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@taimurhafeez
Copy link
Collaborator Author

Hi @taimurhafeez, I don't think it is meaningful to create a dedicate serial test case to check the labels for the ccr. If you really want to cover the test point, I think it is better to adding label queries to 1 existing parallel tests than creating a dedicated serial test
I added to the already TestScanProducesRemediations test but renamed it to TestScanProducesRemediationsAndLabels

@taimurhafeez
Copy link
Collaborator Author

The test case is PASSing succesfully on OCP 4.20 cluster. After running it, one of the worker nodes on my cluster has gone into degraded state though:

$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-2b8ffc374493b63e18f96d2aa594f092   True      False      False      3              3                   3                     0                      102m
worker   rendered-worker-033af400c2a339ce01799d80ce78313c   False     True       True       3              2                   2                     1                      102m

I am not really sure if this is just a coincidence or what is a rootcase yet, so far I just want to post it here so we have a trace that this has happened - in case it happens to someone else as well in the future
@Anna-Koudelkova
It is the same with me on 4.20. I dug further, and a possible reason I found is maybe it uses ocp4-moderate, which affects the cluster state, but the test itself is in parallel?

@rhmdnd
Copy link
Collaborator

rhmdnd commented Dec 2, 2025

Potential transient issue:

 === NAME  TestScanSettingBindingNoStorage
    main_test.go:4184: Expected to find PVC associated with the scan.
2025/12/02 14:16:24 waiting until suite test-custom-rule-with-multiple-inputs-ssb reaches target status 'DONE'. Current status: RUNNING
--- FAIL: TestScanSettingBindingNoStorage (105.17s) 

@taimurhafeez
Copy link
Collaborator Author

Potential transient issue:

 === NAME  TestScanSettingBindingNoStorage
    main_test.go:4184: Expected to find PVC associated with the scan.
2025/12/02 14:16:24 waiting until suite test-custom-rule-with-multiple-inputs-ssb reaches target status 'DONE'. Current status: RUNNING
--- FAIL: TestScanSettingBindingNoStorage (105.17s) 

possibly because I ran recently, and it went through

=== RUN   TestScanProducesRemediationsAndLabels
=== PAUSE TestScanProducesRemediationsAndLabels
=== CONT  TestScanProducesRemediationsAndLabels
2025/12/03 13:33:44 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2025/12/03 13:33:49 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2025/12/03 13:33:54 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2025/12/03 13:33:59 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2025/12/03 13:34:04 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2025/12/03 13:34:09 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2025/12/03 13:34:14 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: AGGREGATING
2025/12/03 13:34:19 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: AGGREGATING
2025/12/03 13:34:24 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: AGGREGATING
2025/12/03 13:34:29 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: AGGREGATING
2025/12/03 13:34:39 ComplianceScan ready (DONE)
2025/12/03 13:34:39 All scans in ComplianceSuite have finished (test-scan-produces-remediations-and-labels)
osdk-e2e-b9e29c29-69f4-4190-b7b2-b6c36fc2663e map[compliance.openshift.io/suite:test-scan-produces-remediations-and-labels]
--- PASS: TestScanProducesRemediationsAndLabels (62.58s)
PASS

@xiaojiey
Copy link
Collaborator

xiaojiey commented Dec 5, 2025

/retest

@taimurhafeez
Copy link
Collaborator Author

/retest

=== RUN   TestScanProducesRemediationsAndLabels
=== PAUSE TestScanProducesRemediationsAndLabels
=== CONT  TestScanProducesRemediationsAndLabels
2025/12/08 16:04:53 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2025/12/08 16:04:58 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2025/12/08 16:05:04 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2025/12/08 16:05:08 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2025/12/08 16:05:13 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2025/12/08 16:05:18 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2025/12/08 16:05:23 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: AGGREGATING
2025/12/08 16:05:28 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: AGGREGATING
2025/12/08 16:05:33 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: AGGREGATING
2025/12/08 16:05:38 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: AGGREGATING
2025/12/08 16:05:48 ComplianceScan ready (DONE)
2025/12/08 16:05:48 All scans in ComplianceSuite have finished (test-scan-produces-remediations-and-labels)
osdk-e2e-053e729a-52f8-4e9c-9c97-6825dc563939 map[compliance.openshift.io/suite:test-scan-produces-remediations-and-labels]
--- PASS: TestScanProducesRemediationsAndLabels (67.72s)
PASS
2025/12/08 16:06:00 cleaning up Profile Bundles

f := framework.Global
bindingName := framework.GetObjNameFromTest(t)
tpName := framework.GetObjNameFromTest(t)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated whitespace change?

Extends: "ocp4-moderate",
},
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated whitespace change?

if err := f.WaitForSuiteScansStatus(f.OperatorNamespace, bindingName, compv1alpha1.PhaseDone, compv1alpha1.ResultNonCompliant); err != nil {
t.Fatal(err)
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated whitespace change?

if err != nil {
t.Fatal(err)
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated whitespace change?

Copy link
Collaborator

@rhmdnd rhmdnd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only minor suggestions inline. Otherwise this is looking good.

@taimurhafeez
Copy link
Collaborator Author

Only minor suggestions inline. Otherwise this is looking good.

@rhmdnd Regarding whitespaces, I was addressing Xiaoji's comment about removing leftover lines, and then I thought it was better to keep them minimized. But now I have addressed this.

Copy link
Collaborator

@rhmdnd rhmdnd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@taimurhafeez
Copy link
Collaborator Author

/retest

Copy link
Collaborator

@rhmdnd rhmdnd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for taking so long to get back to this. I have a couple recommendations on how we can make the assertions a little more robust, and a minor whitespace comment.


return nil
}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unrelated whitespace change?

@openshift-ci openshift-ci bot removed the lgtm label Feb 27, 2026
@openshift-ci-robot
Copy link
Collaborator

@taimurhafeez: This pull request references CMP-3800 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Verify that ComplianceCheckResult objects can be queried using labels:

  • compliance.openshift.io/check-severity
  • compliance.openshift.io/check-status
  • compliance.openshift.io/scan-name
  • compliance.openshift.io/suite

Changes

  • Updated TestScanProducesRemediations to TestScanProducesRemediationsAndLabels in tests/e2e/parallel/main_test.go
  • Add AssertCheckResultByLabel helper in tests/e2e/framework/common.go
  1. Run the test in make e2e-parallel E2E_GO_TEST_FLAGS="-v -run TestScanProducesRemediationsAndLabels"

Tested on OCP 4.21 and expected output:

=== RUN   TestScanProducesRemediationsAndLabels
=== PAUSE TestScanProducesRemediationsAndLabels
=== CONT  TestScanProducesRemediationsAndLabels
2026/03/02 11:53:26 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2026/03/02 11:53:31 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2026/03/02 11:53:36 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2026/03/02 11:53:40 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2026/03/02 11:53:46 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: RUNNING
2026/03/02 11:53:51 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: AGGREGATING
2026/03/02 11:53:56 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: AGGREGATING
2026/03/02 11:54:01 waiting until suite test-scan-produces-remediations-and-labels reaches target status 'DONE'. Current status: AGGREGATING
2026/03/02 11:54:11 ComplianceScan ready (DONE)
2026/03/02 11:54:11 All scans in ComplianceSuite have finished (test-scan-produces-remediations-and-labels)
osdk-e2e-9666bf4d-7b32-4cac-9127-9741af0d0080 map[compliance.openshift.io/suite:test-scan-produces-remediations-and-labels]
--- PASS: TestScanProducesRemediationsAndLabels (52.03s)
PASS

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@taimurhafeez
Copy link
Collaborator Author

taimurhafeez commented Mar 2, 2026

Sorry for taking so long to get back to this. I have a couple recommendations on how we can make the assertions a little more robust, and a minor whitespace comment.

Thanks, Lance, for taking the time to review and useful comments. I have tried to address in my recent commits.

@github-actions
Copy link

github-actions bot commented Mar 2, 2026

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:981-70bd7e087b5991f634157e3e8650e491303c8bf9

Copy link
Collaborator

@rhmdnd rhmdnd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

Thanks for addressing all the comments here. Looks great.

@openshift-ci openshift-ci bot added the lgtm label Mar 2, 2026
@openshift-ci
Copy link

openshift-ci bot commented Mar 2, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rhmdnd, taimurhafeez

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [rhmdnd,taimurhafeez]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@rhmdnd
Copy link
Collaborator

rhmdnd commented Mar 2, 2026

ROSA and serial test failures are unrelated. Attempting a fix for the serial failure (which appears to be transient) in #1100.

Verify that ComplianceCheckResult objects can be queried using labels:
- compliance.openshift.io/check-severity
- compliance.openshift.io/check-status
- compliance.openshift.io/scan-name
- compliance.openshift.io/suite

Add AssertCheckResultByLabel helper to test framework
…nes 3) placed a misplaced bracket 4)Consolidated repetitive label verification code into a single loop structure
…stead of just first and used lighter profile e8
@openshift-ci
Copy link

openshift-ci bot commented Mar 3, 2026

New changes are detected. LGTM label has been removed.

@github-actions
Copy link

github-actions bot commented Mar 3, 2026

🤖 To deploy this PR, run the following command:

make catalog-deploy CATALOG_IMG=ghcr.io/complianceascode/compliance-operator-catalog:981-2341083e285c8e83e835ebf7d2644ee6d96288e6

@openshift-ci
Copy link

openshift-ci bot commented Mar 3, 2026

@taimurhafeez: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-rosa 2341083 link true /test e2e-rosa
ci/prow/e2e-aws-serial 2341083 link true /test e2e-aws-serial
ci/prow/e2e-aws-serial-arm 2341083 link true /test e2e-aws-serial-arm

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@rhmdnd
Copy link
Collaborator

rhmdnd commented Mar 3, 2026

Serial tests are failing becuase of TestProfileBundleDefaultIsKept - which we're attempting to fix in #1100.

This PR only modified the parallel tests, which passed cleanly on both aarch64 and amd64.

@rhmdnd rhmdnd merged commit a2a79b4 into ComplianceAsCode:master Mar 3, 2026
17 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants