feat: Add OpenAPI schema validation for CRD

chmouel · chmouel · commit 1b4ded8ba4ce · 2025-05-12T11:11:00.000+02:00
* Defined OpenAPI v3 schema for the Repository Custom Resource Definition (CRD). This enhanced resource validation, improved client-side tooling like IDE autocompletion, and provided clearer API documentation. * Automated schema generation via a new `make update-schemas` target using `controller-gen`. This ensured schema accuracy and simplified maintenance. * Added a CI job to verify that generated OpenAPI schemas were current. This prevented stale schemas from being merged. * Documented the usage and benefits of the OpenAPI schema for users. This guided users on leveraging schema validation and tooling. * Enhanced `kubectl get repo` output with additional status columns for better observability. JIRA: https://issues.redhat.com/browse/SRVKP-7605 Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
diff --git a/.tekton/linter.yaml b/.tekton/linter.yaml
@@ -90,6 +90,38 @@ spec:
 
                 exit $failed
 
+            - name: check-generated-schemas
+              displayName: "Check generated OpenAPI schemas"
+              image: golang:1.23
+              workingDir: $(workspaces.source.path)
+              env:
+                - name: HUB_TOKEN
+                  valueFrom:
+                    secretKeyRef:
+                      name: "nightly-ci-github-hub-token"
+                      key: "hub-token"
+              script: |
+                set -eu
+                git config --global --add safe.directory $(workspaces.source.path)
+
+                if uname -m | grep -q aarch64; then
+                  arch=arm64
+                else
+                  arch=amd64
+                fi
+
+                # version=$(curl -H "Authorization: Bearer ${HUB_TOKEN}" -L -s https://api.github.com/repos/mikefarah/yq/releases/latest|python3 -c 'import sys, json;dico=json.load(sys.stdin);print(dico["tag_name"])')
+                version=v4.45.2
+                curl -sH "Authorization: Bearer ${HUB_TOKEN}" -L https://github.com/mikefarah/yq/releases/download/${version}/yq_linux_${arch} -o /usr/bin/yq && \
+                  chmod +x /usr/bin/yq
+
+                curl -L -o/usr/bin/kubectl "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/${arch}/kubectl" && \
+                  chmod +x /usr/bin/kubectl
+
+                make update-schemas
+                # check now that the is no changes in config with git
+                git diff --exit-code config/ || { echo "Error: you need to run 'make update-schemas' and commit it."; exit 1; }
+
             - name: conventional-commits
               displayName: "conventional commit check"
               image: registry.access.redhat.com/ubi9/python-312
diff --git a/Makefile b/Makefile
@@ -181,6 +181,11 @@ update-golden: ## run unit tests (updating golden files)
 	@echo "Running unit tests to update golden files..."
 	@./hack/update-golden.sh
 
+.PHONY: update-schemas
+update-schemas: ## update openapi schemas
+	@echo "Running update schemas..."
+	@./hack/update-schemas.sh
+
 .PHONY: generated
 generated: update-golden fumpt ## generate all files that needs to be generated
 
diff --git a/config/300-repositories.yaml b/config/300-repositories.yaml
diff --git a/docs/content/docs/dev/_index.md b/docs/content/docs/dev/_index.md
@@ -195,7 +195,25 @@ We use [golden](https://pkg.go.dev/gotest.tools/v3/golden) files in our tests, f
 make update-golden
 ```
 
-Head over to the [./test/README.md](./test/README.md) for more information on how to update the golden files on the E2E tests.
+Head over to the
+[./test/README.md](https://github.com/openshift-pipelines/pipelines-as-code/blob/main/test/README.md)
+for more information on how to update the golden files on the E2E tests.
+
+## Update OpenAPI Schemas
+
+The CRD schemas are automatically generated from the Go code (generally
+`pkg/apis/pipelinesascode/v1alpha1/types.go`). After modifying
+any type definitions, you'll need to regenerate these schemas to update the CRD
+in `config/300-repositories.yaml`.
+
+When modifying types, ensure the validation logic is appropriate, then run:
+
+```shell
+make update-schemas
+```
+
+There is a PAC CI check that will ensure that the CRD is up to date with the go
+code.
 
 ## Configuring the Pre Push Git checks
 
diff --git a/docs/content/docs/guide/cli.md b/docs/content/docs/guide/cli.md
@@ -1,6 +1,6 @@
 ---
 title: CLI tkn-pac
-weight: 100
+weight: 70
 ---
 # Pipelines-as-Code CLI
 
diff --git a/docs/content/docs/guide/openapi-schema.md b/docs/content/docs/guide/openapi-schema.md
@@ -0,0 +1,136 @@
+---
+title: OpenAPI Schema Validation
+weight: 80
+---
+
+# OpenAPI Schema Validation for Repository CRDs
+
+Pipelines-as-Code provides [OpenAPI](https://www.openapis.org/) schema validation for its Custom Resource
+Definitions (CRDs), which helps in writing the Repository resources. This page
+explains what OpenAPI schemas are, their benefits, and how to leverage them in
+your development environment.
+
+## What is OpenAPI Schema Validation?
+
+OpenAPI schema validation is a mechanism that adds metadata to Kubernetes CRDs, providing:
+
+- **Type information**: Specifies expected data types for fields
+- **Required fields**: Marks which fields must be present
+- **Pattern validation**: Enforces specific formats (e.g., URLs must start with http:// or https://)
+- **Enumeration validation**: Limits fields to a predefined set of values
+- **Field descriptions**: Documents the purpose of each field
+
+When you create or modify a Repository resource, the OpenAPI schema helps
+validate your configuration before it's applied to the cluster, catching errors
+early in the development process.
+
+## Benefits of OpenAPI Schema Validation
+
+Using OpenAPI schemas with Pipelines-as-Code provides numerous advantages:
+
+1. **Early Error Detection**: Identifies configuration errors before applying resources to your cluster
+2. **Improved Documentation**: Field descriptions provide built-in documentation
+3. **Better IDE Support**: Enables rich autocompletion and validation in code editors
+4. **Standardized Formatting**: Ensures your resources follow expected formats
+5. **Self-Documenting API**: Makes it easier to understand the Repository CRD structure
+
+## Using OpenAPI Schemas in VS Code
+
+Visual Studio Code and other modern editors can leverage OpenAPI schemas to
+provide real-time validation, autocompletion, and documentation while editing
+your Repository resources.
+
+### Setting Up VS Code for CRD Validation
+
+![VSCode and openapi Schema](/images/vscode-openapi-schema.png)
+
+1. Install the [Kubernetes](https://marketplace.visualstudio.com/items?itemName=ms-kubernetes-tools.vscode-kubernetes-tools) extension (by Microsoft)
+2. Install the [YAML](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml) extension (by Red Hat)
+
+These extensions will automatically detect and use the OpenAPI schema from your Kubernetes cluster when editing Repository resources.
+
+### Example: Editing a Repository Resource in VS Code
+
+When editing a Repository resource in VS Code with the appropriate extensions, you'll experience:
+
+- **Autocompletion** for fields like `url`, `git_provider`, `settings`, etc.
+- **Validation** for required fields and field formats
+- **Documentation tooltips** when hovering over fields
+- **Enum dropdown suggestions** for fields with predefined values
+
+Here's an example of what you'll see:
+
+```yaml
+apiVersion: pipelinesascode.tekton.dev/v1alpha1
+kind: Repository
+metadata:
+  name: my-repository
+spec:
+  # VS Code will show an error if this required field is missing
+  url: https://github.com/myorg/myrepo
+
+  # Autocompletion will suggest all available fields
+  git_provider:
+    # Dropdown will show available provider types
+    type: github
+
+    # Pattern validation ensures the URL format is correct
+    url: https://api.github.com
+
+    # Documentation tooltips show field descriptions
+    webhook_secret:
+      name: webhook-secret
+      key: token
+
+  # Validation for minimum values
+  concurrency_limit: 5
+
+  settings:
+    # Enum validation limits to specific options
+    pipelinerun_provenance: default_branch
+```
+
+## Other Tools Using OpenAPI Schemas
+
+Beyond VS Code, OpenAPI schemas are useful with:
+
+1. **kubectl**: Validates resources before applying them
+
+   ```bash
+   kubectl create -f my-repository.yaml --validate=true
+   ```
+
+2. **kube-linter**: Validates resources as part of CI/CD pipelines
+
+3. **Kubernetes Dashboard**: Shows field descriptions and validations
+
+4. **OpenAPI documentation generators**: Create API documentation from schemas
+
+## How to Get the OpenAPI Schema
+
+The OpenAPI schema for Repository resources is embedded in the CRD itself. You can view it with:
+
+```bash
+kubectl get crd repositories.pipelinesascode.tekton.dev -o jsonpath='{.spec.versions[0].schema.openAPIV3Schema}' | jq
+```
+
+Or to see all available fields with their descriptions:
+
+```bash
+kubectl explain repository.spec --recursive
+```
+
+## Troubleshooting Schema Validation Errors
+
+If you encounter validation errors when creating or updating a Repository:
+
+1. **Check field types**: Ensure values match expected types
+2. **Verify required fields**: Make sure all required fields are present
+3. **Review pattern validations**: URLs must follow the correct format
+4. **Check enum values**: Some fields accept only specific values
+
+Error messages typically provide specific details about which validation failed, making it easier to correct your resources.
+
+{{< hint info >}}
+Even with client-side validation, the webhook validation in Pipelines-as-Code provides an additional layer of validation for complex logic that cannot be expressed in OpenAPI schemas.
+{{< /hint >}}
diff --git a/docs/content/docs/install/gitlab.md b/docs/content/docs/install/gitlab.md
@@ -119,6 +119,7 @@ $ tkn pac create repo
     url: "https://gitlab.com/group/project"
     git_provider:
       # url: "https://gitlab.example.com/ # Set this if you are using a private GitLab instance
+      type: "gitlab"
       secret:
         name: "gitlab-webhook-config"
         # Set this if you have a different key in your secret
diff --git a/docs/static/images/vscode-openapi-schema.png b/docs/static/images/vscode-openapi-schema.png
diff --git a/hack/update-schemas.sh b/hack/update-schemas.sh
@@ -0,0 +1,156 @@
+#!/usr/bin/env bash
+# Copyright 2025 The Tekton Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Imported from tektoncd/pipeline/hack/update-schemas.sh and vibed/adapted for
+# PAAC.
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+OLDGOFLAGS="${GOFLAGS:-}"
+GOFLAGS=""
+
+SCRIPT_DIR=$(dirname "${BASH_SOURCE[0]}")
+CRD_PATH="${SCRIPT_DIR}/../config"
+API_PATH="${SCRIPT_DIR}/../pkg/apis"
+TEMP_DIR_LOGS=$(mktemp -d)
+
+echo "=== Generating CRD schemas with OpenAPI validation ==="
+
+# List of CRDs to process
+CRD_FILES=(
+  "${CRD_PATH}/300-repositories.yaml"
+  # Add more CRD files here as they are created
+)
+
+for FILENAME in "${CRD_FILES[@]}"; do
+  BASENAME=$(basename "$FILENAME")
+  echo "Generating OpenAPI schema for $FILENAME"
+
+  # Extract group from the CRD file
+  GROUP=$(grep -E '^  group:' $FILENAME)
+  GROUP=${GROUP#"  group: "}
+  API_SUBDIR=${GROUP%".tekton.dev"}
+
+  TEMP_DIR=$(mktemp -d)
+  cp -p $FILENAME $TEMP_DIR/.
+  LOG_FILE=$TEMP_DIR_LOGS/log-schema-generation-$(basename $FILENAME)
+
+  echo "  Processing API group: $GROUP, subdir: $API_SUBDIR"
+
+  counter=0 limit=5
+  while [ "$counter" -lt "$limit" ]; do
+    set +e
+    # Use controller-gen to generate the schema
+    go run sigs.k8s.io/controller-tools/cmd/controller-gen@v0.17.1 \
+      crd:crdVersions=v1 \
+      output:crd:artifacts:config=$TEMP_DIR \
+      paths=$API_PATH/$API_SUBDIR/... >$LOG_FILE 2>&1
+    rc=$?
+    set -e
+
+    if [ $rc -eq 0 ]; then
+      echo "  ✅ Successfully generated schema"
+
+      # Find the auto-generated CRD file
+      if command -v yq >/dev/null 2>&1 && yq --version | grep -q "mikefarah/yq"; then
+        echo "  🔄 Syncing schema from temporary CRD to $BASENAME"
+
+        # Find the auto-generated CRD file in the temp directory
+        AUTO_GENERATED_CRD=$(find $TEMP_DIR -name "${GROUP}_*.yaml")
+
+        if [ -f "$AUTO_GENERATED_CRD" ]; then
+          # Extract schema from auto-generated CRD and apply it to the original CRD
+          yq eval '.spec.versions[0].schema' "$AUTO_GENERATED_CRD" >/tmp/schema.yaml
+          yq eval -i '.spec.versions[0].schema = load("/tmp/schema.yaml")' "$FILENAME"
+          echo "  ✅ Schema successfully synced to $BASENAME"
+
+          # Clean up temporary schema file
+          rm -f /tmp/schema.yaml
+        else
+          echo "  ⚠️  Warning: Auto-generated CRD not found in temporary directory"
+        fi
+      else
+        echo "  ⚠️  Warning: mikefarah/yq not available, cannot automatically sync schema"
+        echo "  Manual action required: Copy schema from auto-generated CRD to $BASENAME"
+      fi
+
+      # Now generate directly to the config directory for the final step
+      # but we'll delete the auto-generated file afterward
+      go run sigs.k8s.io/controller-tools/cmd/controller-gen@v0.17.1 \
+        crd:crdVersions=v1 \
+        output:crd:artifacts:config=$CRD_PATH \
+        paths=$API_PATH/$API_SUBDIR/... >/dev/null 2>&1
+
+      # Find and delete the auto-generated CRD file from the config directory
+      AUTO_GENERATED_CRD=$(find $CRD_PATH -name "${GROUP}_*.yaml")
+      if [ -f "$AUTO_GENERATED_CRD" ]; then
+        echo "  🗑️  Removing auto-generated CRD file: $(basename "$AUTO_GENERATED_CRD")"
+        rm -f "$AUTO_GENERATED_CRD"
+      fi
+
+      break
+    fi
+
+    # Check if we can proceed despite errors
+    if grep -q 'exit status 1' $LOG_FILE; then
+      echo "  ⚠️  Warning: Encountered errors during schema generation"
+      echo "  📝 Check $LOG_FILE for details"
+      # Attempt to extract and display useful error information
+      ERROR_MSG=$(grep -A 3 'Error:' $LOG_FILE 2>/dev/null || echo "Unknown error")
+      echo "  Error summary: $ERROR_MSG"
+      break
+    fi
+
+    counter=$((counter + 1))
+    if [ $counter -eq $limit ]; then
+      echo "  ❌ Failed to generate CRD schema after $limit attempts"
+      echo "  📝 Check $LOG_FILE for details"
+      cat ${LOG_FILE}
+      exit 1
+    fi
+
+    echo "  Retrying (attempt $counter of $limit)..."
+    sleep 1
+  done
+
+  # Validate the generated CRD
+  echo "  Validating generated CRD"
+  if command -v kubectl >/dev/null 2>&1; then
+    # Try using the newer "kubectl server-side validation" approach
+    if kubectl explain crd &>/dev/null; then
+      if kubectl apply --server-side --dry-run=server --validate=strict -f "$FILENAME" >/dev/null 2>&1; then
+        echo "  ✅ CRD validation successful using server-side validation"
+      else
+        echo "  ⚠️  Warning: Server-side CRD validation failed, validation may not be complete"
+      fi
+    else
+      echo "  ⚠️  Warning: kubectl API resources not available, skipping validation"
+    fi
+  else
+    echo "  ⚠️  Warning: kubectl not found, skipping validation"
+  fi
+
+  rm -rf $TEMP_DIR
+done
+
+echo "=== Schema generation complete ==="
+echo "Generated schemas saved to: $CRD_PATH"
+echo "OpenAPI schema has been successfully added to the numbered CRD files"
+echo "Auto-generated reference CRD files have been removed as requested"
+echo "Log files available at: $TEMP_DIR_LOGS"
+
+GOFLAGS="${OLDGOFLAGS}"
diff --git a/pkg/apis/pipelinesascode/v1alpha1/types.go b/pkg/apis/pipelinesascode/v1alpha1/types.go
