Skip to content

Commit 46e38d2

Browse files
haripateMarcinGinsztAlldaamissteawcheang
authored
[ISV-933] Merge the operator-pipeline-images code to operator-pipeline code (#157)
* Initial commit * Add script for building bundle dockerfile The simple CLI tool is added that is capable to generate a bundle dockerfile from bundle metadata. JIRA: ISV-777 Signed-off-by: Ales Raszka <[email protected]> * Add github action to build and push an image Github action is configured to build and push image to quay registry. * Use black formatter in actions Python black formatter is used in github actions to check a python code format. * Add operator-sdk cli tool The operator-sdk tool is used to verify a new bundles in operator pipeline. JIRA: 848 * Add jq and yq tools These tools are used in pipelines to parse yaml and json files. * Add xargs dependency xargs is used in operator CI pipeline. * Add a script to get pertinent OCP version info Refactored the python scripts into a library with lightweight entrypoints to make it easier to test. The existing bundle_dockerfile.py script has a new way to invoke it. It's excluded from code coverage until components of it are moved into the library. * Add a .gitignore * Run tox in GitHub Actions * Load requirements from dedicated files * Cover a couple more failure conditions with unit tests * Add more gitignore rules * Keep utils.py minimal * Add opm CLI * Use a common annotations function in the bundle-dockerfile script * added script that checks the changed directories * added docstrings * removed wrong default param * moved functions to proper file, added unit tests * better function name and docstring * not allowing the ci.yaml to be in operator root directory * using the git url to retrieve org and repo names * removed colorful logs, applying comments * black format * deserializing response after checking http error * changed last occurence of operator_version to bundle_version * Add a test data for result preflight upload This commit contains a test data that can be used to mock a preflight output and upload to Pyxis. * allowing changing the ci.yaml on the operator level * removing repository prefix * change in unit tests * accepting only ssh links to repository * workaround tekton bug, which adds the \n to results * changed underscores to dashes in the script arguments * applying comments * using only strip instead of lstrip and rstrip * Add script for uploading artifacts The script uploads test artifacts using Pyxis API. JIRA: ISV-883 * rebase from main * script to validate operator submission * can't run package as a script * rebase from main * storing regex as rawstring to avoid warning on invalid escape sequence * applying comments * applying comments- spell pull request, allow choice of github organisation * Add manual release action The action creates a new tag, release and container build. The build is pushed to remote registry. JIRA: ISV-920 * Upload pipeline logs The script for uploading results and artifacts is extended to also upload a pipeline logs. JIRA: ISV-927 * Add script to reserve operator package name * rebase from main * script doesn't fail in case of PR with wrong regex * added unit test * black format * fixing changes after rebase * Enable cert auth in uploading script Artifact upload script now support both API key and cert based auth. The auth method is controlled by env variables. JIRA: ISV-974 * added script to get the preflight results * change script name from get_preflight_results to download_artifacts * returning results and logs ids from the script * black * enhanced comments * allowing empty results * black format * downloading only test results * removing unnecessary result and gitignore entry * using the new approach to connect to Pyxis * applying comments * applying comment- empty line at the end of the file * Log association name when operator name is taken by another association This was suggested during the community meeting to help with debugging. Privacy was not a concern since there's not much one can do with just the assocation name without the right API key. * Add script to update github status * Add script to update github status * Add script for opening Github PR The script opens a pull request in a github. The script will be used in CI pipeline. JIRA: ISV-860 * added script to get project related data * added scripts to retrieve the vendor related data and cert project related data * storing whole queried object as result * Add publishing script This commit adds new publishing cli tool that publishes vendor and repository. JIRA: ISV-991 * anything can be bundle version- not only semver * changed one letter to uppercase, black format * updated tests * Accepting https and ssh schema in verify_changed_dirs step (#23) * accepting https and ssh schema in verify_changed_dirs step * black format * Verifying the user submitting the bundle (#24) * verifying the user who submitted the bundle * using the list of contacts instead of file with contacts as an argument * black format * [ISV-749] Update the download_test_results function (#25) * [ISV-914] Update the download_test_results function * [ISV-914] Add github comment on PR script * [ISV-914] Remove github comment on PR script * Remove unused field Co-authored-by: haripate <> * Add script to examine statuses from hydra checklist API * Add script for creation of ContainerImage (#26) * added script to create ContainerImage * black format * applying comments * black format * applying comments- simplifying logic * black format * adding parsed_data to created image * added unit test * making the tag non- floating * removing latest tag via PUT * iterating over the tags a bit more nicely * applying comments * accepting podman results * Modify hydra checklist script to use proxy for preprod Preprod hydra cannot be accessed directly due to the Akamai preprod lockdown, so a proxy needs to be set up. * The ocp-version-info script can now retrieve marketplace indices * installing yamllint in Dockerfile * [ISV-871][ISV-873] Write a script to update the certProject related data (#29) * Write a script to update the certProject related data * Update the certification_time field value * Fix up the function name * Fix up the function name * Fix up the function name * Fix up the function name * Fix up the function name * Fix up the function name Co-authored-by: haripate <> * [ISV-871][ISV-873] Fix up the date time format (#34) * Write a script to update the certProject related data * Update the certification_time field value * Fix up the function name * Fix up the function name * Fix up the function name * Fix up the function name * Fix up the function name * Fix up the function name * Fix up the date time format Co-authored-by: haripate <> * [ISV-871][ISV-873] Fix up the update cert project status function (#35) * Write a script to update the certProject related data * Update the certification_time field value * Fix up the function name * Fix up the function name * Fix up the function name * Fix up the function name * Fix up the function name * Fix up the function name * Fix up the date time format * [ISV-871][ISV-873] Fix up the update cert project status function Co-authored-by: haripate <> * [ISV-993] Add index cli tool (#27) * [ISV-1056] Upload artifacts with file size * Remove unused operator-sdk * [ISV-1219] Add target_url to the set_github_status function (#38) Co-authored-by: haripate <> * Install gnupg to container image Gnugp is used to decrypt project secrets. JIRA: ISV-1205 * Add script to call IBM webhook to trigger marketplace replication * properly logging the results * added the code documentation * Update way to get relatedImages * Code review comments: use organization as arg instead of git-repo-url * splitting the validate submission script into the components of it * black format * Rename 'production' to 'prod' To align recent changes in pipeline repo we have to rename prod environment naming. * Add origin-clients dep to run oc * proper head and base (#43) * Revert "proper head and base (#43)" (#44) This reverts commit cb49a43. * Fix up the pipeline errors * Fix up the pipeline errors * Fix up the pipeline errors * Fix up the pipeline errors * Fix up the pipeline errors * Fix up the pipeline errors * Update the merged repo format * Update the merged repo format * Update the merged repo format * Update the merged repo format * Update the merged repo format * Update the merged repo format * Test on pipeline * Test on pipeline * Test on pipeline * Testing the merged repo * Testing the merged repo * Final test of the merged repo * Final test of the merged repo Co-authored-by: MarcinGinszt <[email protected]> Co-authored-by: Ales Raszka <[email protected]> Co-authored-by: Alex Misstear <[email protected]> Co-authored-by: [email protected] <[email protected]> Co-authored-by: Wai Cheang <[email protected]> Co-authored-by: Jan Koscielniak <[email protected]> Co-authored-by: haripate <>
1 parent 5c8d2ef commit 46e38d2

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

73 files changed

+6671
-1
lines changed

‎.coveragerc‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
[run]
2+
source = operator-pipeline-images/operatorcert
3+
omit = operator-pipeline-images/operatorcert/webhook/*
4+
5+
[report]
6+
omit = operator-pipeline-images/operatorcert/entrypoints/*

‎.github/workflows/main-image.yml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
name: Build and Push Image
2+
on: # yamllint disable-line rule:truthy
3+
- push
4+
- pull_request
5+
6+
jobs:
7+
test-lint:
8+
name: Run unit tests and linters
9+
runs-on: ubuntu-latest
10+
steps:
11+
- uses: actions/checkout@v2
12+
- uses: fedora-python/[email protected]
13+
with:
14+
tox_env: black,test
15+
dnf_install: krb5-devel krb5-workstation
16+
17+
build:
18+
name: Build and push image
19+
runs-on: ubuntu-20.04
20+
21+
steps:
22+
- uses: actions/checkout@v2
23+
24+
- name: Build Image
25+
id: build-image
26+
uses: redhat-actions/buildah-build@v2
27+
with:
28+
image: operator-pipelines-images
29+
tags: latest ${{ github.sha }}
30+
dockerfiles: |
31+
./operator-pipeline-images/Dockerfile
32+
33+
- name: Push To quay.io
34+
id: push-to-quay
35+
uses: redhat-actions/push-to-registry@v2
36+
with:
37+
image: ${{ steps.build-image.outputs.image }}
38+
tags: ${{ steps.build-image.outputs.tags }}
39+
registry: quay.io/redhat-isv
40+
username: ${{ secrets.REGISTRY_USERNAME }}
41+
password: ${{ secrets.REGISTRY_PASSWORD }}
42+
if: ${{ github.event_name == 'push' }}
43+
44+
- name: Print image url
45+
run: echo "Image pushed to ${{ steps.push-to-quay.outputs.registry-paths }}"
46+
if: ${{ github.event_name == 'push' }}

‎.github/workflows/release-image.yml‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
name: Release image
2+
3+
on: workflow_dispatch
4+
5+
jobs:
6+
build:
7+
name: Build and push image
8+
runs-on: ubuntu-20.04
9+
10+
steps:
11+
- uses: actions/checkout@v2
12+
13+
- name: Bump version and push tag
14+
id: tag_version
15+
uses: mathieudutour/[email protected]
16+
with:
17+
github_token: ${{ secrets.GITHUB_TOKEN }}
18+
19+
- name: Create a GitHub release
20+
uses: actions/create-release@v1
21+
env:
22+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
23+
with:
24+
tag_name: ${{ steps.tag_version.outputs.new_tag }}
25+
release_name: Release ${{ steps.tag_version.outputs.new_tag }}
26+
body: ${{ steps.tag_version.outputs.changelog }}
27+
28+
- name: Build Image
29+
id: build-image
30+
uses: redhat-actions/buildah-build@v2
31+
with:
32+
image: operator-pipelines-images
33+
tags: latest ${{ github.sha }} ${{ steps.tag_version.outputs.new_tag }}
34+
dockerfiles: |
35+
./operator-pipeline-images/Dockerfile
36+
37+
- name: Push To quay.io
38+
id: push-to-quay
39+
uses: redhat-actions/push-to-registry@v2
40+
with:
41+
image: ${{ steps.build-image.outputs.image }}
42+
tags: ${{ steps.build-image.outputs.tags }}
43+
registry: quay.io/redhat-isv
44+
username: ${{ secrets.REGISTRY_USERNAME }}
45+
password: ${{ secrets.REGISTRY_PASSWORD }}
46+
47+
- name: Print image url
48+
run: echo "Image pushed to ${{ steps.push-to-quay.outputs.registry-paths }}"

‎.gitignore‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,139 @@
22
ssh-secret.yml
33
registry-secret.yml
44
kubeconfig
5+
# results
6+
test_results.json
7+
results_exists
8+
test_logs_id
9+
test_result_id
10+
bundle_name
11+
bundle_version
12+
13+
# Byte-compiled / optimized / DLL files
14+
__pycache__/
15+
*.py[cod]
16+
*$py.class
17+
18+
# C extensions
19+
*.so
20+
21+
# Distribution / packaging
22+
.Python
23+
build/
24+
develop-eggs/
25+
dist/
26+
downloads/
27+
eggs/
28+
.eggs/
29+
lib/
30+
lib64/
31+
parts/
32+
sdist/
33+
var/
34+
wheels/
35+
pip-wheel-metadata/
36+
share/python-wheels/
37+
*.egg-info/
38+
.installed.cfg
39+
*.egg
40+
MANIFEST
41+
42+
# PyInstaller
43+
# Usually these files are written by a python script from a template
44+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
45+
*.manifest
46+
*.spec
47+
48+
# Installer logs
49+
pip-log.txt
50+
pip-delete-this-directory.txt
51+
52+
# Unit test / coverage reports
53+
htmlcov/
54+
.tox/
55+
.nox/
56+
.coverage
57+
.coverage.*
58+
.cache
59+
nosetests.xml
60+
coverage.xml
61+
*.cover
62+
.hypothesis/
63+
.pytest_cache/
64+
65+
# Translations
66+
*.mo
67+
*.pot
68+
69+
# Django stuff:
70+
./*.log
71+
./*.log.*
72+
local_settings.py
73+
db.sqlite3
74+
db.sqlite3-journal
75+
76+
# Flask stuff:
77+
instance/
78+
.webassets-cache
79+
80+
# Scrapy stuff:
81+
.scrapy
82+
83+
# Sphinx documentation
84+
docs/_build/
85+
86+
# PyBuilder
87+
target/
88+
89+
# Jupyter Notebook
90+
.ipynb_checkpoints
91+
92+
# IPython
93+
profile_default/
94+
ipython_config.py
95+
96+
# pyenv
97+
.python-version
98+
99+
# pipenv
100+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
101+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
102+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
103+
# install all needed dependencies.
104+
#Pipfile.lock
105+
106+
# celery beat schedule file
107+
celerybeat-schedule
108+
109+
# SageMath parsed files
110+
*.sage.py
111+
112+
# Environments
113+
.env
114+
.venv
115+
env/
116+
venv/
117+
ENV/
118+
env.bak/
119+
venv.bak/
120+
121+
# Spyder project settings
122+
.spyderproject
123+
.spyproject
124+
125+
# Rope project settings
126+
.ropeproject
127+
128+
# mkdocs documentation
129+
/site
130+
131+
# mypy
132+
.mypy_cache/
133+
.dmypy.json
134+
dmypy.json
135+
136+
# Pyre type checker
137+
.pyre/
5138

6139
# VSCode project settings
7140
.vscode/

‎README.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -171,3 +171,22 @@ tkn pipeline start operator-release-pipeline \
171171
--workspace name=ocp-registry-kubeconfig,secret=ocp-registry-kubeconfig \
172172
--showlog
173173
```
174+
175+
# operator-pipelines-images
176+
Container images containing the set of tools for Partner Operator Bundle [certification pipelines](https://github.com/redhat-openshift-ecosystem/operator-pipelines).
177+
178+
## Development
179+
180+
To install the python package in a development environment, run:
181+
182+
```bash
183+
pip install ".[dev]"
184+
```
185+
186+
To test the scripts with the pipelines, see [local-dev.md](docs/local-dev.md).
187+
188+
To run unit tests and code style checkers:
189+
190+
```bash
191+
tox
192+
```

‎docs/local-dev.md‎

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,4 +51,32 @@ oc adm policy add-scc-to-user privileged -z pipeline
5151
5252
```bash
5353
--pod-template templates/crc-pod-template.yml
54-
```
54+
```
55+
# Local development of the operator-pipelines-images
56+
57+
Development of the Python script doesn't demand any sophisticated strategy. However, before adding the script to release branch,
58+
in order to test them they must run within the pipeline.
59+
60+
## Prerequisites
61+
62+
1. Install [Buildah](https://github.com/containers/buildah/blob/main/install.md)
63+
2. Setup [operator-pipelines](https://github.com/redhat-openshift-ecosystem/operator-pipelines/blob/main/docs/local-dev.md)
64+
65+
## Initial Setup
66+
1. If you are adding a new script- don't forget to add the entrypoint to setup.py
67+
2. Build the image containing the script via Buildah
68+
```bash
69+
buildah bud
70+
```
71+
3. Push the image to registry, eg. Quay.io.
72+
```bash
73+
buildah push <image signature- output of build step> <path in registry>
74+
```
75+
This step may require login, eg.
76+
```bash
77+
buildah login quay.io
78+
```
79+
4. In [operator-pipelines](https://github.com/redhat-openshift-ecosystem/operator-pipelines), change the base image of script that
80+
was changed, to the one pushed in the previous step.
81+
82+
5. Run the pipeline and test if the results are as expected.

‎operator-pipeline-images/.dockerignore‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
# Ignoring git and cache folders
2+
.git
3+
.github
4+
.cache
5+
6+
# Ignoring all the markdown and class files
7+
*.md
8+
docs
9+
10+
#Ignoring tekton pipeine and ansible tasks
11+
templates
12+
ansible

‎operator-pipeline-images/Dockerfile‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
FROM registry.fedoraproject.org/fedora:34
2+
3+
LABEL description="Cli tools for operator certification pipeline"
4+
LABEL summary="This image contains tools required for operator bundle certification pipeline."
5+
6+
ARG USER_UID=1000
7+
8+
USER root
9+
10+
# setup certificates
11+
COPY operator-pipeline-images/certs/* /etc/pki/ca-trust/source/anchors/
12+
RUN /usr/bin/update-ca-trust
13+
14+
# This is just a temporary workaround until we figure out how to
15+
# override CA bundle in OCP
16+
RUN cp /etc/pki/tls/certs/ca-bundle.crt /etc/pki/tls/certs/custom-ca-bundle.crt
17+
18+
ENV REQUESTS_CA_BUNDLE="/etc/pki/tls/certs/custom-ca-bundle.crt"
19+
20+
RUN dnf update -y && \
21+
dnf install -y \
22+
findutils \
23+
git \
24+
gcc \
25+
gnupg2 \
26+
jq \
27+
krb5-devel \
28+
krb5-workstation \
29+
yamllint \
30+
openssl-devel \
31+
origin-clients \
32+
pinentry \
33+
pip \
34+
python3-devel && \
35+
dnf clean all
36+
37+
COPY operator-pipeline-images/config/krb5.conf /etc/krb5.conf
38+
39+
# Install opm CLI
40+
RUN curl -LO https://github.com/operator-framework/operator-registry/releases/download/v1.17.5/linux-amd64-opm && \
41+
chmod +x linux-amd64-opm && \
42+
mv linux-amd64-opm /usr/local/bin/opm
43+
44+
RUN useradd -ms /bin/bash -u "${USER_UID}" user
45+
46+
WORKDIR /home/user
47+
48+
COPY ./operator-pipeline-images ./
49+
50+
RUN pip3 install .
51+
52+
# set dir ownership
53+
RUN chgrp -R 0 /home/user /etc/passwd
54+
RUN chmod -R g=u /home/user /etc/passwd
55+
56+
USER "${USER_UID}"

‎operator-pipeline-images/MANIFEST.in‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
include requirements.txt
2+
include requirements-dev.txt

0 commit comments

Comments
 (0)