-
Notifications
You must be signed in to change notification settings - Fork 105
[WIP]Refactor test/extended to use OpenShift Tests Extension (OTE) framework #568
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
wangke19
wants to merge
6
commits into
openshift:main
Choose a base branch
from
wangke19:separate-ote-scaffold
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
daabf23
Refactor test/extended to use OpenShift Tests Extension (OTE) framework
wangke19 6d73626
go mod vendor in tests-extension
wangke19 471e01c
Fix tests-extension build by running from correct module directory
wangke19 61bb4bf
Rename openshift_apiserver.go to oas.go
wangke19 7cc0c61
Fix gofmt formatting in main.go
wangke19 5d4266b
Fix ARTIFACT_DIR default assignment in run-suite target
wangke19 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
The diff you're trying to view is too large. We only load the first 3000 changed files.
There are no files selected for viewing
1 change: 1 addition & 0 deletions
1
.openshift-tests-extension/openshift_payload_openshift-apiserver.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| [] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| # Binaries for programs and plugins | ||
| *.exe | ||
| *.exe~ | ||
| *.dll | ||
| *.so | ||
| *.dylib | ||
|
|
||
| # Test binary, built with `go test -c` | ||
| *.test | ||
|
|
||
| # Output of the go coverage tool, specifically when used with LiteIDE | ||
| *.out | ||
|
|
||
| # Dependency directories (remove the comment below to include it) | ||
| # vendor/ | ||
|
|
||
| # Go workspace file | ||
| go.work | ||
|
|
||
| # Build artifacts | ||
| bin/ | ||
| .openshift-tests-extension/ | ||
|
|
||
| # IDE files | ||
| .vscode/ | ||
| .idea/ | ||
| *.swp | ||
| *.swo | ||
| *~ | ||
|
|
||
| # OS generated files | ||
| .DS_Store | ||
| .DS_Store? | ||
| ._* | ||
| .Spotlight-V100 | ||
| .Trashes | ||
| ehthumbs.db | ||
| Thumbs.db |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,138 @@ | ||
| # Get the directory where this Makefile is, so we can use it below for including | ||
| DIR := $(strip $(shell dirname $(realpath $(lastword $(MAKEFILE_LIST))))) | ||
|
|
||
| # Definitions for the extended tests | ||
|
|
||
| GO_PKG_NAME := github.com/openshift-eng/openshift-tests-extension | ||
|
|
||
| GIT_COMMIT := $(shell git rev-parse --short HEAD 2>/dev/null || echo 'unknown') | ||
| BUILD_DATE := $(shell date -u +'%Y-%m-%dT%H:%M:%SZ') | ||
| GIT_TREE_STATE := $(shell if git rev-parse --git-dir > /dev/null 2>&1; then if git diff --quiet; then echo clean; else echo dirty; fi; else echo unknown; fi) | ||
|
|
||
| LDFLAGS := -X '$(GO_PKG_NAME)/pkg/version.CommitFromGit=$(GIT_COMMIT)' \ | ||
| -X '$(GO_PKG_NAME)/pkg/version.BuildDate=$(BUILD_DATE)' \ | ||
| -X '$(GO_PKG_NAME)/pkg/version.GitTreeState=$(GIT_TREE_STATE)' | ||
|
|
||
| REPO_ROOT := $(shell git rev-parse --show-toplevel) | ||
| METADATA := $(REPO_ROOT)/.openshift-tests-extension/openshift_payload_openshift-apiserver.json | ||
|
|
||
| .PHONY: help | ||
| help: #HELP Display essential help. | ||
| @awk 'BEGIN {FS = ":[^#]*#HELP"; printf "\nUsage:\n make \033[36m<target>\033[0m\n\n"} /^[a-zA-Z_0-9-]+:.*#HELP / { printf " \033[36m%-17s\033[0m %s\n", $$1, $$2 } ' $(MAKEFILE_LIST) | ||
|
|
||
| #SECTION Tests | ||
| TOOLS_BIN_DIR := $(CURDIR)/bin | ||
|
|
||
| .PHONY: run-suite #HELP Run tests by suite name and generate JUnit report (e.g., make run-suite SUITE=openshift/openshift-apiserver/all ARTIFACT_DIR=/tmp/artifacts) | ||
| run-suite: build #HELP Run tests by suite name and generate JUnit report | ||
| @if [ -z "$(SUITE)" ]; then \ | ||
| echo "Error: SUITE variable is required. Available suites:"; \ | ||
| echo " - openshift/openshift-apiserver/all"; \ | ||
| echo " - openshift/openshift-apiserver/conformance/parallel"; \ | ||
| echo " - openshift/openshift-apiserver/conformance/serial"; \ | ||
| echo " - openshift/openshift-apiserver/optional/slow"; \ | ||
| echo ""; \ | ||
| echo "Usage: make run-suite SUITE=openshift/openshift-apiserver/all [ARTIFACT_DIR=/tmp/artifacts]"; \ | ||
| exit 1; \ | ||
| fi | ||
| @ARTIFACT_DIR_FINAL=$${ARTIFACT_DIR:-$(CURDIR)}; \ | ||
| echo "Using ARTIFACT_DIR: $$ARTIFACT_DIR_FINAL"; \ | ||
| mkdir -p $$ARTIFACT_DIR_FINAL; \ | ||
| $(TOOLS_BIN_DIR)/openshift-apiserver-tests-ext run-suite $(SUITE) -j $$ARTIFACT_DIR_FINAL/junit_$(shell date +%Y%m%d-%H%M%S).xml | ||
|
|
||
| #SECTION Development | ||
| .PHONY: verify #HELP To verify the code | ||
| verify: tidy fmt vet lint | ||
|
|
||
| .PHONY: tidy #HELP Run go mod tidy. | ||
| tidy: | ||
| go mod tidy | ||
|
|
||
| .PHONY: fmt | ||
| fmt: #HELP Run go fmt against code. | ||
| go fmt ./... | ||
|
|
||
| .PHONY: vet | ||
| vet: #HELP Run go vet against code. | ||
| go vet ./... | ||
|
|
||
| .PHONY: lint | ||
| lint: #HELP Run golangci linter. | ||
| @echo "Linting not configured for tests-extension" | ||
|
|
||
| .PHONY: fix-lint | ||
| fix-lint: #HELP Fix lint issues | ||
| @echo "Linting not configured for tests-extension" | ||
|
|
||
| # GO_COMPLIANCE_POLICY="exempt_all" must only be used for test related binaries. | ||
| # It prevents various FIPS compliance policies from being applied to this compilation. | ||
| # Do not set globally. | ||
| .PHONY: build | ||
| build: #HELP Build the extended tests binary | ||
| @mkdir -p $(TOOLS_BIN_DIR) | ||
| GO_COMPLIANCE_POLICY="exempt_all" go build -ldflags "$(LDFLAGS)" -o $(TOOLS_BIN_DIR)/openshift-apiserver-tests-ext ./cmd/... | ||
|
|
||
| .PHONY: update-metadata | ||
| update-metadata: build #HELP Build and run 'update-metadata' to generate test metadata | ||
| $(TOOLS_BIN_DIR)/openshift-apiserver-tests-ext update --component openshift:payload:openshift-apiserver | ||
| $(MAKE) clean-metadata | ||
|
|
||
|
|
||
| # Ensure TestID is unique over time. | ||
| # TestID is built over Product:Type:ComponentName:TestDescription | ||
| # (i.e. openshift:payload:openshift-apiserver:TestName) | ||
| # Details: | ||
| # - https://github.com/openshift/enhancements/blob/master/enhancements/testing/openshift-tests-extension.md#test-id | ||
| # - https://github.com/openshift-eng/ci-test-mapping | ||
| #────────────────────────────────────────────────────────────── | ||
| # How to rename a test? | ||
| # 1. Run: make list-test-names | ||
| # 2. Find the current full test name (e.g. "[sig-abc] My test does XYZ") | ||
| # 3. Add a Ginkgo label: ginkgo.Label("original-name:[sig-abc] My test does XYZ"). if there is existing original-name label, please do not update the label again and keep it unchanged. | ||
| # 4. Change the test name string and run: make build-update | ||
| # **Example** | ||
| # It("should pass a renamed sanity check", | ||
| # Label("original-name:[sig-openshift-apiserver] OLMv1 should pass a trivial sanity check"), | ||
| # func(ctx context.Context) { | ||
| # Expect(len("test")).To(BeNumerically(">", 0)) | ||
| # }) | ||
| # Note: You only add the label once. Do not update it after future renames. | ||
| #────────────────────────────────────────────────────────────── | ||
| # How to delete a test? | ||
| # 1. Run: make list-test-names | ||
| # 2. In main.go add: | ||
| # ext.IgnoreObsoleteTests( | ||
| # "[sig-openshift-apiserver] My removed test name", | ||
| # ) | ||
| # 3. Delete the test code in your suite file (e.g. oas.go) | ||
| # 4. Run: make build-update. | ||
| # This will regenerate the metadata without the test entry. | ||
| #──────────────────────────────────────────────────────────────────── | ||
| .PHONY: build-update | ||
| build-update: build update-metadata #HELP Build and update metadata and sanitize output | ||
|
|
||
| .PHONY: clean | ||
| clean: #HELP Remove build artifacts | ||
| rm -rf $(TOOLS_BIN_DIR) | ||
|
|
||
| #SECTION Metadata | ||
|
|
||
| .PHONY: list-test-names | ||
| list-test-names: build #HELP Show current full test names | ||
| @$(TOOLS_BIN_DIR)/openshift-apiserver-tests-ext list -o names | ||
|
|
||
| # Remove 'codeLocations' to avoid absolute paths like: | ||
| # "/Users/$(USER)/go/src/.../oas.go:12" | ||
| # These are machine-specific and make the metadata non-idempotent. | ||
| # More info: https://issues.redhat.com/browse/TRT-2186 | ||
| .PHONY: clean-metadata | ||
| clean-metadata: #HELP Remove 'codeLocations' from metadata JSON | ||
| @echo "Cleaning metadata (removing codeLocations)..." | ||
| @jq 'map(del(.codeLocations))' $(METADATA) > $(METADATA).tmp && mv $(METADATA).tmp $(METADATA) | ||
|
|
||
| .PHONY: verify-metadata #HELP To verify that the metadata was properly updated | ||
| verify-metadata: update-metadata | ||
| @if ! git diff --exit-code $(METADATA); then \ | ||
| echo "ERROR: Metadata is out of date. Please run 'make build-update' and commit the result."; \ | ||
| exit 1; \ | ||
| fi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.