Simplify PgSTAC Bootstrap Process (#208)

* feat: Add PgSTAC settings SQL file and update bootstrap process

- Introduced a new SQL file for PgSTAC settings to improve maintainability.
- Updated the PgSTAC bootstrap process to utilize Kubernetes jobs for database initialization and sample data loading.
- Replaced custom Python script with the pypgstac migrate command for schema initialization.
- Enhanced configuration structure for better clarity and control over sample data loading.
- Deprecated the legacy environment variable approach in favor of a more structured settings approach.

* feat: Add EOAPI Helm Chart README and streamline PgSTAC bootstrap process

* feat: Enhance debugging output for failed service boot and test failures in CI workflow

* fix: reduce timeout for service boot watch to 2 minutes

* feat: update pgstac image name to use pgstac-pypgstac variant

* fix: update log extraction for pgstac-migrate pod in CI workflow

* fix: update wait command to reference pgstac-migrate job in deployment.yaml

* fix: disable continue-on-error for test runner step in CI workflow

* fix: enhance pgstac-migrate job monitoring and increase timeout for service boot watch

* fix: add log extraction for pgstac-migrate pod in CI workflow

* fix: add error handling to pgstac-migrate job script

* fix: improve database readiness check and add error handling in pgstac-migrate job

* fix: add debugging output for environment variables in pgstac-migrate job

* fix: update postgres user privileges and clarify comments in values.yaml

* fix: configure Patroni for schema permissions and update PostgreSQL settings

* fix: update schema permissions for eoapi user in values.yaml

* fix: update PostgreSQL schema privileges for eoapi user in values.yaml

* fix: update postgrescluster options to SUPERUSER for enhanced privileges

* fix: update database initialization SQL and modify postgrescluster options for enhanced privileges

* fix: simplify database initialization SQL configuration in values.yaml

* fix: correct database options formatting in values.yaml

* fix: add CREATE privilege for eoapi user on public schema in initdb.sql

* fix: update privileges for eoapi user in initdb.sql

* fix: update initdb.sql to set default privileges for eoapi user

* fix: streamline database initialization and add verification in pgstac bootstrap job

* fix: enhance database initialization verification in pgstac bootstrap job

* fix: enhance search_path verification for PgSTAC functionality in database initialization

* fix: improve database initialization verification by checking eoapi role and search_path

* fix: ensure immediate exit on command failure in PgSTAC migration script

* fix: add database owner assignment and remove redundant initialization verification in pgstac bootstrap job

* fix: streamline PgSTAC migration job by removing redundant PostGIS extension registration and permission settings

* fix: add debugging steps for pgstac-migrate job failures and allow continuation on error

* fix: enhance logging extraction for pgstac-migrate job by implementing multiple pod retrieval methods

* fix: update initdb ConfigMap to include role creation and privileges for pgstac

* fix: ensure exit status is captured after helm test execution

* fix: add initdb ConfigMap for PostgreSQL setup with necessary roles and extensions

* fix: update helm render/install step to include an ID for better tracking

* fix: update database connection settings in pgstac bootstrap ConfigMap and job for improved flexibility

* fix: remove redundant pod log extraction methods for pgstac-migrate

* fix: remove debugging environment variable prints and unused initContainer from pgstac-load-samples job

* fix: remove unnecessary sleep step from helm-tests workflow

Author: emmanuelmathot

.github/workflows/helm-tests.yml

@@ -74,7 +74,9 @@ jobs:
             oci://registry.developers.crunchydata.com/crunchydata/pgo \
             --version ${{ env.PGO_VERSION }}
 
-      - name: helm render/install eoapi templates
+      - id: helm-render-install-eoapi-templates
+        name: helm render/install eoapi templates
+        continue-on-error: true
         run: |
           export GITSHA='${{github.sha}}'
 
@@ -87,18 +89,53 @@ jobs:
             -f ./eoapi/test-k3s-unittest-values.yaml \
             ./eoapi
 
-      - name: sleep for 10s seconds while services boot
-        shell: bash
-        run: sleep 10s
+          exit $?
 
-#      - name: Setup upterm session
-#        uses: lhotari/action-upterm@v1
+      - name: debug pgstac-migrate job failure
+        if: steps.helm-render-install-eoapi-templates.outcome == 'failure'
+        continue-on-error: true
+        run: |
+          echo "Extracting pgstac-migrate job info and logs for debugging..."
+          
+          # Get job details
+          echo "===== pgstac-migrate Job Details ====="
+          kubectl get job pgstac-migrate -o yaml || echo "Could not get pgstac-migrate job details"
+          
+          # Get pod details
+          echo "===== pgstac-migrate Pod Details ====="
+          kubectl get pods -l app=pgstac-migrate --all-namespaces || echo "Could not find pgstac-migrate pods"
+          
+          # Extract logs from pgstac-migrate pod(s)
+          echo "===== pgstac-migrate Pod Logs ====="
+          echo "Looking for completed pods from pgstac-migrate job..."
+          COMPLETED_PODS=$(kubectl get pods --selector=job-name=pgstac-migrate --field-selector=status.phase=Succeeded,status.phase=Failed -o jsonpath='{.items[*].metadata.name}' 2>/dev/null)
+          if [ -n "$COMPLETED_PODS" ]; then
+            echo "Found completed pods from pgstac-migrate job. Extracting logs from each:"
+            for POD in $COMPLETED_PODS; do
+              echo "--- Logs from completed pod $POD ---"
+              kubectl logs pod/$POD || echo "Could not get logs from pod $POD"
+            done
+          else
+            echo "No completed pods found for pgstac-migrate job"
+          fi
+          
+          # Get details about the database pods/services
+          echo "===== Database Pod/Service Details ====="
+          # Find database service
+          kubectl get svc | grep "db" || echo "Could not find database services"
+          # Find database pods
+          kubectl get pods | grep "db-" || echo "Could not find database pods"
+          
+          # Check for any events related to the job or pods
+          echo "===== Related Kubernetes Events ====="
+          kubectl get events | grep -E "pgstac|db" || echo "No relevant events found"
 
       - id: watchservices
         name: watch services boot
-        timeout-minutes: 10
+        timeout-minutes: 3
         continue-on-error: true
         run: |
+          # Now wait for services to boot up
           while [[ -z "$(kubectl get pod  | grep "^raster-$RELEASE_NAME-.*$" | cut -d' ' -f1 | xargs -I{}  kubectl logs pod/{} | grep "GET /.*/healthz" | head -n 1)" ]]; do
             echo "still waiting for raster service to start..."
             sleep 1
@@ -124,14 +161,6 @@ jobs:
           echo "===== Pod Status ====="
           kubectl get pods
           
-          # Extract logs from database pod
-          echo "===== Database Pod Logs ====="
-          kubectl get pod | grep "^db-$RELEASE_NAME" | cut -d' ' -f1 | xargs -I{} kubectl logs pod/{} --tail=100 || echo "Could not get database logs"
-          
-          # Extract logs from pgstacbootstrap pod
-          echo "===== PGSTACBootstrap Pod Logs ====="
-          kubectl get pod | grep "^pgstacbootstrap-$RELEASE_NAME" | cut -d' ' -f1 | xargs -I{} kubectl logs pod/{} --tail=100 || echo "Could not get pgstacbootstrap logs"
-          
           # Extract logs from raster pod init container (wait-for-pgstacbootstrap)
           echo "===== Raster Pod Init Container Logs (wait-for-pgstacbootstrap) ====="
           kubectl get pod | grep "^raster-$RELEASE_NAME" | cut -d' ' -f1 | xargs -I{} kubectl logs pod/{} -c wait-for-pgstacbootstrap --tail=100 || echo "Could not get raster init container logs"
@@ -161,7 +190,7 @@ jobs:
 
       - name: run the tests
         id: testrunner
-        continue-on-error: true
+        # continue-on-error: true
         run: |
           kubectl get svc --all-namespaces
           kubectl get ingress --all-namespaces -o jsonpath='{range .items[0]}kubectl describe ingress {.metadata.name} -n {.metadata.namespace}{end}' | sh
@@ -192,14 +221,6 @@ jobs:
           echo "===== Pod Status ====="
           kubectl get pods
           
-          # Extract logs from database pod
-          echo "===== Database Pod Logs ====="
-          kubectl get pod | grep "^db-$RELEASE_NAME" | cut -d' ' -f1 | xargs -I{} kubectl logs pod/{} --tail=100 || echo "Could not get database logs"
-          
-          # Extract logs from pgstacbootstrap pod
-          echo "===== PGSTACBootstrap Pod Logs ====="
-          kubectl get pod | grep "^pgstacbootstrap-$RELEASE_NAME" | cut -d' ' -f1 | xargs -I{} kubectl logs pod/{} --tail=100 || echo "Could not get pgstacbootstrap logs"
-          
           # Extract logs from raster pod init container (wait-for-pgstacbootstrap)
           echo "===== Raster Pod Init Container Logs (wait-for-pgstacbootstrap) ====="
           kubectl get pod | grep "^raster-$RELEASE_NAME" | cut -d' ' -f1 | xargs -I{} kubectl logs pod/{} -c wait-for-pgstacbootstrap --tail=100 || echo "Could not get raster init container logs"
@@ -225,4 +246,4 @@ jobs:
 
       - name: helm uninstall eoapi templates
         run: |
-          helm uninstall $RELEASE_NAME
+          helm uninstall $RELEASE_NAME

helm-chart/eoapi/README.md

@@ -0,0 +1,141 @@
+# EOAPI Helm Chart
+
+This Helm chart deploys the EOAPI (Earth Observation API) stack, which includes STAC API, raster tile services, vector tile services, and a multidimensional data service.
+
+## Overview
+
+The chart sets up:
+
+- A PostgreSQL database with PostGIS and PgSTAC extensions
+- STAC API service for metadata discovery and search
+- Titiler for raster tile services
+- TIPG for vector tile services
+- Optional multidimensional data service
+
+## Prerequisites
+
+- Kubernetes 1.16+
+- Helm 3.0+
+- PV provisioner support in the underlying infrastructure
+- CrunchyData Postgres Operator (for the PostgreSQL database)
+
+## Installation
+
+```bash
+# Install Postgres Operator first
+helm install --set disable_check_for_upgrades=true pgo oci://registry.developers.crunchydata.com/crunchydata/pgo
+
+# Then install eoapi
+helm install eoapi ./eoapi
+```
+
+## Configuration
+
+The chart can be configured via `values.yaml`. See the chart's `values.yaml` file for all available options and detailed descriptions.
+
+Key configuration sections:
+
+```yaml
+# Services to enable
+apiServices:
+  - raster
+  - stac
+  - vector
+  # - multidim (disabled by default)
+
+# Ingress configuration
+ingress:
+  enabled: true
+  className: "nginx"
+  # ...
+
+# Database configuration
+postgrescluster:
+  enabled: true
+  # ...
+```
+
+## PgSTAC Bootstrap Process
+
+The chart includes a streamlined process for initializing and setting up the PgSTAC database.
+
+### PgSTAC Bootstrap Overview
+
+The setup process consists of two main jobs:
+
+1. **pgstac-migrate job**: Runs the pypgstac migrate command to initialize the database schema, applies settings, and sets necessary permissions.
+2. **pgstac-load-samples job**: (Optional) Loads sample STAC data only when sample loading is enabled.
+
+### Improvements in PgSTAC Bootstrap
+
+- Replaced custom Python script with pypgstac migrate command
+- Moved SQL settings to a dedicated SQL file for better maintainability
+- Separated sample data loading into an optional job
+- Uses standard PostgreSQL environment variables
+- Ensures the process remains idempotent for safe re-runs
+
+### PgSTAC Directory Structure
+
+The codebase has been reorganized to separate different types of files:
+
+- `initdb-data/settings/`: Contains configuration settings like the PgSTAC settings SQL file
+- `initdb-data/samples/`: Contains sample data files that are loaded only when sample loading is enabled
+
+### PgSTAC Configuration
+
+- Enable/disable the setup process through `pgstacBootstrap.enabled`
+- Control sample data loading: 
+  - New approach: `pgstacBootstrap.settings.loadSamples` (recommended)
+  - Legacy approach: `pgstacBootstrap.settings.envVars.LOAD_FIXTURES` (deprecated)
+
+Example configuration:
+
+```yaml
+pgstacBootstrap:
+  enabled: true
+  settings:
+    # General configuration options
+    loadSamples: true      # Set to false to disable sample data loading
+    
+    resources:
+      requests:
+        cpu: "512m"
+        memory: "1024Mi"
+      limits:
+        cpu: "512m"
+        memory: "1024Mi"
+```
+
+## Services
+
+### STAC API
+
+The STAC API service provides a standardized way to search and discover geospatial data.
+
+### Raster Services (Titiler)
+
+Provides dynamic tiling for raster data through the TiTiler implementation.
+
+### Vector Services (TIPG)
+
+Provides vector tile services for PostGIS data through the TIPG implementation.
+
+### Multidimensional Services (Optional)
+
+Provides services for multidimensional data (time series, etc.).
+
+## Persistence
+
+The chart uses PostgreSQL for data persistence. Make sure to configure appropriate storage for production use.
+
+## Upgrading
+
+When upgrading the chart, consider any changes to values.yaml and migrations that might need to be applied.
+
+## Uninstallation
+
+```bash
+helm delete eoapi
+```
+
+Note that PVs may need to be manually deleted if you want to remove all data.

helm-chart/eoapi/initdb-data/pgstac-setup.py

@@ -0,0 +1,12 @@
+-- Apply pgstac settings
+DELETE FROM pgstac.pgstac_settings WHERE name = 'context';
+INSERT INTO pgstac.pgstac_settings (name, value) VALUES ('context', 'auto');
+
+DELETE FROM pgstac.pgstac_settings WHERE name = 'context_estimated_count';
+INSERT INTO pgstac.pgstac_settings (name, value) VALUES ('context_estimated_count', '100000');
+
+DELETE FROM pgstac.pgstac_settings WHERE name = 'context_estimated_cost';
+INSERT INTO pgstac.pgstac_settings (name, value) VALUES ('context_estimated_cost', '100000');
+
+DELETE FROM pgstac.pgstac_settings WHERE name = 'context_stats_ttl';
+INSERT INTO pgstac.pgstac_settings (name, value) VALUES ('context_stats_ttl', '1 day');

helm-chart/eoapi/initdb-data/my_data.sql renamed to helm-chart/eoapi/initdb-data/samples/my_data.sql

@@ -1,5 +1,4 @@
 {{- if .Values.db.enabled }}
----
 apiVersion: v1
 kind: ConfigMap
 metadata: