Add ATEX testing to the upstream CI workflows #14203

ggbecker · 2025-12-02T11:38:04Z

Description:

Add ATEX testing to the upstream CI workflows
It posts the resulting tests link as a comment to the pull request when it finishes.

Rationale:

This aims to replace existing testing farm individual checks to a centralized ATEX execution that can run all CI upstream tests for PRs and manage them accordingly.
The PR as of now, runs only a single STIG hardening test on Centos Stream 8/9/10 to be served as a proof of concept. We should eventually extend to include more tests similarly as the current upstream CI.

ggbecker · 2025-12-02T14:49:06Z

From: https://github.com/ComplianceAsCode/content/actions/runs/19858710726/job/56915038821?pr=14203

   File "/usr/local/lib/python3.13/site-packages/atex/provisioner/testingfarm/api.py", line 507, in reserve
    if self.api.whoami()["token"]["ranch"] == "public":
       ~~~~~~~~~~~~~~~^^
  File "/usr/local/lib/python3.13/site-packages/atex/provisioner/testingfarm/api.py", line 122, in whoami
    raise ValueError("whoami() requires an auth token")
ValueError: whoami() requires an auth token

I'm afraid the token will only be allowed to be used when we merge the pull request

https://github.com/ComplianceAsCode/content/pull/14203/files#diff-9581118f6672453f95900179c8eccc554c1b111682dab06bd16317909fdaf295R109

The same code with the same token is working fine on my fork: ggbecker#41

Mab879 · 2025-12-02T19:33:19Z

tests/submit_results_to_testing_farm.py

@@ -0,0 +1,88 @@
+#!/usr/bin/python3


Suggested change

#!/usr/bin/python3

#!/usr/bin/env python3

Mab879 · 2025-12-02T19:33:58Z

tests/run_tests_testingfarm.py

+    platform_name = f"cs{args.os_major_version}@{args.arch}"
+
+    # Hardware requirements for Testing Farm
+    # if args.arch == "x86_64":


Dead code should be removed, we can keep in history if needed

Mab879 · 2025-12-02T19:34:18Z

tests/run_tests_testingfarm.py

@@ -0,0 +1,128 @@
+#!/usr/bin/python3


Suggested change

#!/usr/bin/python3

#!/usr/bin/env python3

Mab879 · 2025-12-02T19:36:45Z

tests/run_tests_testingfarm.py

+logger = logging.getLogger("ATEX")
+
+# Parse command-line arguments
+parser = argparse.ArgumentParser(description="Run tests on Testing Farm using atex")


So both of these script work, but are not structured like our other scripts (main method, parse_args, etc). Any reason for that? I would prefer that these scripts follow format of our other scripts.

Mab879 · 2025-12-02T19:39:42Z

.github/workflows/atex.yaml

+            libxslt-devel python3-setuptools gawk
+
+      - name: Install Python dependencies
+        run: pip install pcre2==0.4.0 -r requirements.txt -r test-requirements.txt


Do we just want add pcre2 to test-requirements.txt?

Mab879 · 2025-12-02T19:44:19Z

.github/workflows/atex.yaml

+
+      - name: Download test results - CentOS Stream 9
+        if: always()
+        uses: actions/download-artifact@v4


Use hashes, check latest version

Mab879 · 2025-12-02T19:44:24Z

.github/workflows/atex.yaml

+
+      - name: Download test results - CentOS Stream 10
+        if: always()
+        uses: actions/download-artifact@v4


Use hashes, check latest version

Mab879 · 2025-12-02T19:44:52Z

.github/workflows/atex.yaml

+
+      - name: Checkout ATEX HTML viewer
+        if: always()
+        uses: actions/checkout@v4


Use hashes, check latest version

tests/submit_results_to_testing_farm.py

Mab879 · 2025-12-02T19:56:34Z

tests/submit_results_to_testing_farm.py

+
+# do faster queries than the default 30 secs, because we don't track
+# many dozens of requests, just one
+class FastRequest(api.Request):


Consider adding a docstring for why this class is needed.

Mab879 · 2025-12-02T20:39:03Z

.github/workflows/atex-test.yaml

+
+      - name: Run tests on Testing Farm
+        env:
+          TESTING_FARM_API_TOKEN: ${{ secrets.TESTING_FARM_API_TOKEN }}


IIRC, this will never be available since we are using pull_request.

We might need break this out into two jobs. One that builds using pull_request, then a second one that uses workflow_run to trigger the tests.

Also added restrictions to token permissions as suggested by github.

Use env in the shebang for python scripts. Use fixed hash for github actions to prevent supply chain attacks. Remove dead code. Add pcre2 to the requirements.txt file.

…e repo.

ggbecker · 2025-12-03T13:44:52Z

I've addressed all the feedback provided and split the jobs into two workflows, one with the workflow_run as suggested by @Mab879 . Let's see how it behaves.

openshift-ci · 2025-12-03T14:07:18Z

@ggbecker: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-openshift-platform-compliance	`ccbbfcb`	link	true	`/test e2e-aws-openshift-platform-compliance`
ci/prow/e2e-aws-openshift-node-compliance	`ccbbfcb`	link	true	`/test e2e-aws-openshift-node-compliance`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Mab879 · 2025-12-03T18:43:44Z

.github/workflows/atex-build.yaml

+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4


use hash, this applies other places in the PR, but I will not repeat.

Mab879 · 2025-12-03T18:46:27Z

.github/workflows/atex-test.yaml

+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          run-id: ${{ github.event.workflow_run.id }}
+          name: pr-info-9


Should this also be using the matrix? We do save this for all versions of RHEL.

Mab879 · 2025-12-03T18:47:10Z

.github/workflows/atex-build.yaml

+            libxslt-devel python3-setuptools gawk
+
+      - name: Install Python dependencies
+        run: pip install pcre2==0.4.0 -r requirements.txt -r test-requirements.txt


pcre2 should be in the requirements.txt file

Mab879 · 2025-12-03T18:49:07Z

.github/workflows/atex-test.yaml

+      pr_sha: ${{ steps.get_pr.outputs.pr_sha }}
+    steps:
+      - name: Download PR info
+        uses: actions/download-artifact@018cc2cf5baa6db3ef3c5f8a56943fffe632ef53 # v4


Checkout is on v6 let's upgrade to the latest. Please double check all versions of actions in this PR.

Mab879 · 2025-12-03T18:53:43Z

.github/workflows/atex-test.yaml

+  contents: read
+  actions: read
+  pull-requests: write
+  issues: write


Is issues: write really needed?

Add new atex test workflow.

a2466a0

ggbecker added this to the 0.1.80 milestone Dec 2, 2025

ggbecker requested review from Mab879, jan-cerny, matusmarhefka and vojtapolasek as code owners December 2, 2025 11:38

ggbecker added the Test Suite Update in Test Suite. label Dec 2, 2025

Refactor ATEX workflow code.

301dab3

ggbecker force-pushed the atex-workflow-tests-3 branch from f1dae23 to 301dab3 Compare December 2, 2025 12:32

Mab879 requested changes Dec 2, 2025

View reviewed changes

Mab879 reviewed Dec 2, 2025

View reviewed changes

ggbecker mentioned this pull request Dec 3, 2025

[duplicate] Add ATEX testing to the upstream CI workflows #14209

Open

ggbecker added 3 commits December 3, 2025 13:07

Split workflows into separate jobs for better secret handling.

90de204

Also added restrictions to token permissions as suggested by github.

Apply suggestions.

6dfbfad

Use env in the shebang for python scripts. Use fixed hash for github actions to prevent supply chain attacks. Remove dead code. Add pcre2 to the requirements.txt file.

Refactor testingfarm scripts to follow pattern from other files in th…

ccbbfcb

…e repo.

Mab879 self-assigned this Dec 3, 2025

Mab879 reviewed Dec 3, 2025

View reviewed changes

Add ATEX testing to the upstream CI workflows #14203

Are you sure you want to change the base?

Add ATEX testing to the upstream CI workflows #14203

Conversation

ggbecker commented Dec 2, 2025

Description:

Rationale:

Uh oh!

ggbecker commented Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggbecker commented Dec 3, 2025

Uh oh!

openshift-ci bot commented Dec 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ggbecker commented Dec 2, 2025 •

edited

Loading