Skip to content
6 changes: 5 additions & 1 deletion common/.github/workflows/pattern-sh-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,14 @@ jobs:
# Fedora is not an option yet
os: [ubuntu-latest, ubuntu-22.04]
runs-on: ${{ matrix.os }}
permissions:
contents: read

steps:
- name: Checkout Repository
uses: actions/checkout@v4
uses: actions/checkout@v5
with:
persist-credentials: false

- name: Install Podman on Ubuntu
if: contains(matrix.os, 'ubuntu')
Expand Down
12 changes: 5 additions & 7 deletions common/.github/workflows/superlinter.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,19 +9,22 @@ jobs:
name: Super linter
# Set the agent to run on
runs-on: ubuntu-latest
permissions:
contents: read

steps:
- name: Checkout Code
uses: actions/checkout@v4
uses: actions/checkout@v5
with:
# Full git history is needed to get a proper list of changed files within `super-linter`
fetch-depth: 0
persist-credentials: false

################################
# Run Linter against code base #
################################
- name: Lint Code Base
uses: super-linter/super-linter/slim@v7
uses: super-linter/super-linter/slim@ffde3b2b33b745cb612d787f669ef9442b1339a6 # v8.1.0
env:
VALIDATE_ALL_CODEBASE: true
DEFAULT_BRANCH: main
Expand All @@ -33,12 +36,7 @@ jobs:
VALIDATE_JSCPD: false
VALIDATE_JSON_PRETTIER: false
VALIDATE_MARKDOWN_PRETTIER: false
VALIDATE_KUBERNETES_KUBECONFORM: false
VALIDATE_PYTHON_PYLINT: false
VALIDATE_SHELL_SHFMT: false
VALIDATE_YAML: false
VALIDATE_YAML_PRETTIER: false
# VALIDATE_DOCKERFILE_HADOLINT: false
# VALIDATE_MARKDOWN: false
# VALIDATE_NATURAL_LANGUAGE: false
# VALIDATE_TEKTON: false
83 changes: 44 additions & 39 deletions common/Changes.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,119 +2,119 @@

## Sep 24, 2024

* Ansible has been moved out of the common code tree, you must use a clustergroup chart that is >= 0.9.1
- Ansible has been moved out of the common code tree, you must use a clustergroup chart that is >= 0.9.1

## Sep 6, 2024

* Most charts have been removed from the tree. To get the charts you now have to point to them
- Most charts have been removed from the tree. To get the charts you now have to point to them

## Sep 25, 2023

* Upgraded ESO to v0.9.5
- Upgraded ESO to v0.9.5

## Aug 17, 2023

* Introduced support for multisource applications via .chart + .chartVersion
- Introduced support for multisource applications via .chart + .chartVersion

## Jul 8, 2023

* Introduced a default of 20 for sync failures retries in argo applications (global override via global.options.applicationRetryLimit
- Introduced a default of 20 for sync failures retries in argo applications (global override via global.options.applicationRetryLimit
and per-app override via .syncPolicy)

## May 22, 2023

* Upgraded ESO to 0.8.2
* *Important* we now use the newly blessed sso config for argo. This means that gitops < 1.8 are *unsupported*
- Upgraded ESO to 0.8.2
- *Important* we now use the newly blessed sso config for argo. This means that gitops < 1.8 are *unsupported*

## May 18, 2023

* Introduce a EXTRA_HELM_OPTS env variable that will be passed to the helm invocations
- Introduce a EXTRA_HELM_OPTS env variable that will be passed to the helm invocations

## April 21, 2023

* Added labels and annotation support to namespaces.yaml template
- Added labels and annotation support to namespaces.yaml template

## Apr 11, 2023

* Apply the ACM ocp-gitops-policy everywhere but the hub
- Apply the ACM ocp-gitops-policy everywhere but the hub

## Apr 7, 2023

* Moved to gitops-1.8 channel by default (stable is unmaintained and will be dropped starting with ocp-4.13)
- Moved to gitops-1.8 channel by default (stable is unmaintained and will be dropped starting with ocp-4.13)

## March 20, 2023

* Upgraded ESO to 0.8.1
- Upgraded ESO to 0.8.1

## February 9, 2023

* Add support for /values-<platform>.yaml and for /values-<platform>-<clusterversion>.yaml
- Add support for /values-<platform>.yaml and for /values-<platform>-<clusterversion>.yaml

## January 29, 2023

* Stop extracting the HUB's CA via an imperative job running on the imported cluster.
- Stop extracting the HUB's CA via an imperative job running on the imported cluster.
Just use ACM to push the HUB's CA out to the managed clusters.

## January 23, 2023

* Add initial support for running ESO on ACM-imported clusters
- Add initial support for running ESO on ACM-imported clusters

## January 18, 2023

* Add validate-schema target
- Add validate-schema target

## January 13, 2023

* Simplify the secrets paths when using argo hosted sites
- Simplify the secrets paths when using argo hosted sites

## January 10, 2023

* vaultPrefixes is now optional in the v2 secret spec and defaults to ["hub"]
- vaultPrefixes is now optional in the v2 secret spec and defaults to ["hub"]

## December 9, 2022

* Dropped insecureUnsealVaultInsideCluster (and file_unseal) entirely. Now
- Dropped insecureUnsealVaultInsideCluster (and file_unseal) entirely. Now
vault is always unsealed via a cronjob in the cluster. It is recommended to
store the imperative/vaultkeys secret offline securely and then delete it.

## December 8, 2022

* Removed the legacy installation targets:
- Removed the legacy installation targets:
`deploy upgrade legacy-deploy legacy-upgrade`
Patterns must now use the operator-based installation

## November 29, 2022

* Upgraded vault-helm to 0.23.0
* Enable vault-ssl by default
- Upgraded vault-helm to 0.23.0
- Enable vault-ssl by default

## November 22, 2022

* Implemented a new format for the values-secret.yaml. Example can be found in examples/ folder
* Now the order of values-secret file lookup is the following:
- Implemented a new format for the values-secret.yaml. Example can be found in examples/ folder
- Now the order of values-secret file lookup is the following:
1. ~/values-secret-<patternname>.yaml
2. ~/values-secret.yaml
3. <patterngitrepo>/values-secret.yaml.template
* Add support for ansible vault encrypted values-secret files. You can now encrypt your values-secret file
1. ~/values-secret.yaml
1. <patterngitrepo>/values-secret.yaml.template
- Add support for Ansible vault encrypted values-secret files. You can now encrypt your values-secret file
at rest with `ansible-vault encrypt ~/values-secret.yaml`. When running `make load-secrets` if an encrypted
file is encountered the user will be prompted automatically for the password to decrypt it.

## November 6, 2022

* Add support for /values-<CloudPlatform>-<clusterGroup>.yaml (e.g. /values-AWS-group-one.yaml)
- Add support for /values-<CloudPlatform>-<clusterGroup>.yaml (e.g. /values-AWS-group-one.yaml)

## October 28, 2022

* Updated vault helm chart to v0.22.1 and vault containers to 1.12.0
- Updated vault helm chart to v0.22.1 and vault containers to 1.12.0

## October 25, 2022

* Updated External Secrets Operator to v0.6.0
* Moved to -UBI based ESO containers
- Updated External Secrets Operator to v0.6.0
- Moved to -UBI based ESO containers

## October 13, 2022

* Added global.clusterVersion as a new helm variable which represents the OCP
- Added global.clusterVersion as a new helm variable which represents the OCP
Major.Minor cluster version. By default now a user can add a
values-<ocpversion>-<clustergroup>.yaml file to have specific cluster version
overrides (e.g. values-4.10-hub.yaml). Will need Validated Patterns Operator >= 0.0.6
Expand All @@ -123,20 +123,25 @@

## October 4, 2022

* Extended the values-secret.yaml file to support multiple vault paths and re-wrote
- Extended the values-secret.yaml file to support multiple vault paths and re-wrote
the push_secrets feature as python module plugin. This requires the following line
in a pattern's ansible.cfg's '[defaults]' stanza:

`library=~/.ansible/plugins/modules:./ansible/plugins/modules:./common/ansible/plugins/modules:/usr/share/ansible/plugins/modules`

## October 3, 2022

* Restore the ability to install a non-default site: `make TARGET_SITE=mysite install`
* Revised tests (new output and filenames, requires adding new result files to Git)
* ACM 2.6 required for ACM-based managed sites
* Introduced global.clusterDomain template variable (without the `apps.` prefix)
* Removed the ability to send specific charts to another cluster, use hosted argo sites instead
* Added the ability to have the hub host `values-{site}.yaml` for spoke clusters.
- Restore the ability to install a non-default site: `make TARGET_SITE=mysite install`

- Revised tests (new output and filenames, requires adding new result files to Git)

- ACM 2.6 required for ACM-based managed sites

- Introduced global.clusterDomain template variable (without the `apps.` prefix)

- Removed the ability to send specific charts to another cluster, use hosted argo sites instead

- Added the ability to have the hub host `values-{site}.yaml` for spoke clusters.

The following example would deploy the namespaces, subscriptions, and
applications defined in `values-group-one.yaml` to the `perth` cluster
Expand Down
30 changes: 17 additions & 13 deletions common/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ endif
# Set this to true if you want to skip any origin validation
DISABLE_VALIDATE_ORIGIN ?= false
ifeq ($(DISABLE_VALIDATE_ORIGIN),true)
VALIDATE_ORIGIN :=
VALIDATE_ORIGIN :=
else
VALIDATE_ORIGIN := validate-origin
endif
Expand All @@ -31,9 +31,8 @@ TARGET_BRANCH ?= $(shell git rev-parse --abbrev-ref HEAD)
#default to the branch remote
TARGET_ORIGIN ?= $(shell git config branch.$(TARGET_BRANCH).remote)

# This is to ensure that whether we start with a git@ or https:// URL, we end up with an https:// URL
# This is because we expect to use tokens for repo authentication as opposed to SSH keys
TARGET_REPO=$(shell git ls-remote --get-url --symref $(TARGET_ORIGIN) | sed -e 's/.*URL:[[:space:]]*//' -e 's%^git@%%' -e 's%^https://%%' -e 's%:%/%' -e 's%^%https://%')
# The URL for the configured origin (could be HTTP/HTTPS/SSH)
TARGET_REPO_RAW := $(shell git ls-remote --get-url --symref $(TARGET_ORIGIN))

UUID_FILE ?= ~/.config/validated-patterns/pattern-uuid
UUID_HELM_OPTS ?=
Expand All @@ -50,13 +49,22 @@ TOKEN_SECRET ?=
TOKEN_NAMESPACE ?=

ifeq ($(TOKEN_SECRET),)
HELM_OPTS=-f values-global.yaml --set main.git.repoURL="$(TARGET_REPO)" --set main.git.revision=$(TARGET_BRANCH) $(TARGET_SITE_OPT) $(UUID_HELM_OPTS) $(EXTRA_HELM_OPTS)
# SSH agents are not created for public repos (repos with no secret token) by the patterns operator so we convert to HTTPS
TARGET_REPO := $(shell echo "$(TARGET_REPO_RAW)" | sed 's/^git@\(.*\):\(.*\)/https:\/\/\1\/\2/')
SECRET_OPTS :=
else
# When we are working with a private repository we do not escape the git URL as it might be using an ssh secret which does not use https://
TARGET_CLEAN_REPO=$(shell git ls-remote --get-url --symref $(TARGET_ORIGIN))
HELM_OPTS=-f values-global.yaml --set main.tokenSecret=$(TOKEN_SECRET) --set main.tokenSecretNamespace=$(TOKEN_NAMESPACE) --set main.git.repoURL="$(TARGET_CLEAN_REPO)" --set main.git.revision=$(TARGET_BRANCH) $(TARGET_SITE_OPT) $(UUID_HELM_OPTS) $(EXTRA_HELM_OPTS)
TARGET_REPO := $(TARGET_REPO_RAW)
SECRET_OPTS := --set main.tokenSecret=$(TOKEN_SECRET) --set main.tokenSecretNamespace=$(TOKEN_NAMESPACE)
endif

HELM_OPTS := -f values-global.yaml \
--set main.git.repoURL="$(TARGET_REPO)" \
--set main.git.revision=$(TARGET_BRANCH) \
$(SECRET_OPTS) \
$(TARGET_SITE_OPT) \
$(UUID_HELM_OPTS) \
$(EXTRA_HELM_OPTS)

# Helm does the right thing and fetches all the tags and detects the newest one
PATTERN_INSTALL_CHART ?= oci://quay.io/hybridcloudpatterns/pattern-install

Expand Down Expand Up @@ -138,8 +146,6 @@ token-kubeconfig: ## Create a local ~/.kube/config with password (not usually ne

##@ Validation Tasks

# We only check the remote ssh git branch's existance if we're not running inside a container
# as getting ssh auth working inside a container seems a bit brittle
# If the main repoUpstreamURL field is set, then we need to check against
# that and not target_repo
.PHONY: validate-origin
Expand Down Expand Up @@ -250,16 +256,14 @@ super-linter: ## Runs super linter locally
-e VALIDATE_JSCPD=false \
-e VALIDATE_JSON_PRETTIER=false \
-e VALIDATE_MARKDOWN_PRETTIER=false \
-e VALIDATE_KUBERNETES_KUBECONFORM=false \
-e VALIDATE_PYTHON_PYLINT=false \
-e VALIDATE_SHELL_SHFMT=false \
-e VALIDATE_TEKTON=false \
-e VALIDATE_YAML=false \
-e VALIDATE_YAML_PRETTIER=false \
$(DISABLE_LINTERS) \
-v $(PWD):/tmp/lint:rw,z \
-w /tmp/lint \
ghcr.io/super-linter/super-linter:slim-v7
ghcr.io/super-linter/super-linter@sha256:6c71bd17ab38ceb7acb5b93ef72f5c2288b5456a5c82693ded3ee8bb501bba7f # slim-v8.1.0

.PHONY: deploy upgrade legacy-deploy legacy-upgrade
deploy upgrade legacy-deploy legacy-upgrade:
Expand Down
8 changes: 4 additions & 4 deletions common/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
This is the `main` branch of common and it assumes that the pattern is fully
multisource (meaning that any used charts from VP is actually referenced from
either a helm chart repository or quay repository). I.e. there are no helm
charts contained in this branch of common and there is no ansible code neither.
charts contained in this branch of common and there is no Ansible code neither.

The helm charts now live in separate repositories under the VP
[organization](https://github.com/validatedpatterns) on GitHub. The repositories are:
Expand All @@ -19,10 +19,10 @@ The helm charts now live in separate repositories under the VP
- acm-chart
- letsencrypt-chart

The ansible bits live in this [repository](https://github.com/validatedpatterns/rhvp.cluster_utils)
The Ansible bits live in this [repository](https://github.com/validatedpatterns/rhvp.cluster_utils)

In order to be able to use this "slimmed-down" main branch of common you *must*
use a 0.9.* clustergroup-chart that. Add the following to your `values-global.yaml`:
use a 0.9.\* clustergroup-chart that. Add the following to your `values-global.yaml`:

```yaml
main:
Expand All @@ -48,4 +48,4 @@ git merge -s subtree -Xtheirs -Xsubtree=common common-upstream/main

## Secrets

There are two different secret formats parsed by the ansible bits. Both are documented [here](https://github.com/validatedpatterns/common/tree/main/ansible/roles/vault_utils/README.md)
There are two different secret formats parsed by the Ansible bits. Both are documented [in the vault role](https://github.com/validatedpatterns/rhvp.cluster_utils/blob/main/roles/vault_utils/README.md)