test: added data and actual export to integration tests.

Author: pantierra

.github/workflows/ci.yml

@@ -30,10 +30,12 @@ jobs:
         with:
           k3s-channel: latest
       - name: Run integration test
-        run: ./scripts/integration-test.sh
+        run: ./scripts/integration-test.sh --cleanup
+        env:
+          USE_EXISTING_CLUSTER: "true"
       - name: Debug on failure
         if: failure()
         run: kubectl get all -n test
       - name: Cleanup
         if: always()
-        run: k3d cluster delete pgstac-test || true
+        run: ./scripts/integration/cleanup.sh --namespace || true

charts/pgstac-geoparquet-exporter/templates/cronjob.yaml

@@ -99,6 +99,10 @@ spec:
             - name: AWS_S3_ENDPOINT
               value: {{ . | quote }}
             {{- end }}
+            {{- with $.Values.stacApiUrl }}
+            - name: STAC_API_URL
+              value: {{ . | quote }}
+            {{- end }}
             {{- with $.Values.storage.region }}
             - name: AWS_REGION
               value: {{ . | quote }}

charts/pgstac-geoparquet-exporter/values.yaml

@@ -10,6 +10,11 @@ nameOverride: ""
 fullnameOverride: ""
 namespaceOverride: ""
 
+# STAC API URL for dynamic link injection (optional)
+# When set, enables the stac-geoparquet library to inject links into items
+# retrieved from pgSTAC. This is recommended as pgSTAC doesn't store links.
+stacApiUrl: ""
+
 # Export job configurations
 jobs:
   complete:

scripts/README.md

@@ -1,30 +1,96 @@
-# Integration Tests
+# Scripts
 
-Kubernetes integration test using k3d for pgstac-geoparquet-exporter.
+## integration-test.sh
 
-## Requirements
+End-to-end integration test validating the complete export pipeline from pgSTAC to GeoParquet.
 
-- Docker
-- [k3d](https://k3d.io)
-- kubectl
-- helm
-
-## Run
+### Quick Start
 
 ```bash
 ./scripts/integration-test.sh
+
+# Optional: customize cluster/namespace
+CLUSTER_NAME=my-cluster NAMESPACE=my-ns ./scripts/integration-test.sh
+```
+
+**Requirements**: kubectl, helm, python3 (pip). k3d auto-installed if no cluster exists.
+
+### What It Tests
+
+1. **Infrastructure**: k3d cluster, PostgreSQL Operator, pgSTAC, MinIO, Helm chart
+2. **Data Loading**: Creates test collection + 3 STAC items via pypgstac
+3. **Export Jobs**: Triggers complete export (and incremental - expected to fail)
+4. **Validation**: Reads GeoParquet files with PyArrow, validates schema and row count
+
+### Expected Output
+
+```
+✓ ALL TESTS PASSED SUCCESSFULLY
+════════════════════════════════════════════════════════════
+  ✓ CronJobs created: 2
+  ✓ STAC items loaded: 3
+  ✓ Complete export: SUCCESS
+  ✓ Geoparquet validation: PASSED
+════════════════════════════════════════════════════════════
 ```
 
-## What it does
+### Test Data
 
-1. Creates k3d cluster
-2. Installs PostgreSQL Operator + pgSTAC database
-3. Deploys MinIO (S3)
-4. Installs helm chart
-5. Verifies CronJobs are created
+- **Collection**: `test-collection` (STAC 1.0.0, global extent, 2020-2023)
+- **Items**: 3 items with polygon geometries, datetime properties, and assets
 
-## Cleanup
+### Cleanup
 
 ```bash
 k3d cluster delete pgstac-test
 ```
+
+### Troubleshooting
+
+**Data loading fails**
+```bash
+kubectl get pods -n test -l postgres-operator.crunchydata.com/cluster=pgstac
+```
+
+**Export job fails**
+```bash
+kubectl logs -n test job/test-complete-export
+```
+
+**Validation fails**
+```bash
+# Check MinIO
+kubectl get pods -n test -l app=minio
+
+# Access MinIO UI
+kubectl port-forward -n test svc/minio 9000:9000
+# Open http://localhost:9000 (minioadmin/minioadmin)
+```
+
+**Python deps fail**: Test skips validation and reports partial success
+
+### Manual Inspection
+
+Keep cluster running and inspect:
+
+```bash
+# View resources
+kubectl get all -n test
+
+# Trigger manual export
+kubectl create job manual-$(date +%s) \
+  --from=cronjob/exporter-pgstac-geoparquet-exporter-complete -n test
+
+# View logs
+kubectl logs -n test -l app=geoparquet-export --tail=100
+```
+
+### CI/CD
+
+```yaml
+- name: Integration Tests
+  run: ./scripts/integration-test.sh
+  timeout-minutes: 15
+```
+
+**Runtime**: ~5-8 minutes