test: S1 GRD E2E validation + workflow improvements #11
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Three Jupyter notebooks demonstrating GeoZarr data access and pyramid features: 01_quickstart.ipynb - Load GeoZarr from S3 with embedded STAC metadata - Visualize RGB composites - Inspect geospatial properties 02_pyramid_performance.ipynb - Benchmark tile serving with/without pyramids - Measure observed 3-5× speedup at zoom 6-10 - Calculate storage tradeoffs (33% overhead) 03_multi_resolution.ipynb - Access individual pyramid levels (0-3) - Compare sizes (4.7MB → 72KB reduction) - Explore quality vs size tradeoffs These notebooks help users understand the pipeline outputs and evaluate pyramid benefits for their use cases. Still evolving as we refine the conversion process and gather production feedback.
Replace inline bash script in workflows/amqp-publish-once.yaml with scripts/publish_amqp.py. Script is now included in Docker image, eliminating need for runtime pip installs and curl downloads. Changes: - Add scripts/publish_amqp.py with routing key templates and retry - Update workflows/amqp-publish-once.yaml to use pre-built image - Add workflows/ directory to docker/Dockerfile - Add tests/unit/test_publish_amqp.py with pytest-mock
20 tests: pattern matching, S1/S2 configs, CLI output formats
Tests asset priority logic (product > zarr > any .zarr) and error handling for missing or malformed STAC items.
Tests subprocess execution, timeout handling, error cases, and CLI options including file output and verbose mode.
Measures load time and dataset metrics for performance comparison. Outputs JSON results with speedup factor and format recommendations.
- S1 quickstart example script for local testing - Comprehensive guide covering S1 vs S2 differences - Documents collection registry, preview generation, workflow params
- Add show-parameters step displaying full workflow config in UI - Add step headers (1/4, 2/4, etc) to all pipeline stages - Add progress indicators and section dividers for better readability - Add workflow metadata labels (collection, item-id) for filtering - Fix sensor event binding (rabbitmq-geozarr/geozarr-events) - Add S1 E2E test job (amqp-publish-s1-e2e.yaml) Argo UI now shows: • Full payload/parameters in dedicated initial step • Clear step numbers and progress for each stage • Final URLs for STAC item and S3 output • Better context during long-running conversions
Complete validation report showing: - Successful S1 GRD to GeoZarr conversion - 21-minute workflow execution (30k x 15k resolution) - 6-level multiscale pyramids for VV/VH polarizations - STAC registration with preview links - UI enhancements validated in Argo - Collection registry parameters documented
- Fix sys.path in test_publish_amqp.py from parent.parent to parent.parent.parent - Update S1 spatial_chunk test expectations from 2048 to 4096 - Aligns with code changes in get_conversion_params.py
- Remove test_real_stac_api_connection (only checked HTTP 200, no logic) - Remove unused os import - Test had external dependency, was flaky, redundant with mocked tests
- Format long argparse description lines for readability - No functional changes, purely formatting
- Set archiveLogs: false for immediate log visibility via kubectl - Change convert-geozarr from script to container template for stdout logs - Reduce memory request to 6Gi (limit 10Gi) for better cluster scheduling - Add Dask parallel processing info in comments - Simplify show-parameters to basic output Fixes 30-60s log delay in Argo UI. Logs now visible via kubectl immediately.
- Add run-s1-test.yaml for direct kubectl submission - Update amqp-publish-s1-e2e.yaml with optimized test parameters - Use S1A item from Oct 3 for consistent testing
- Add WORKFLOW_SUBMISSION_TESTING.md with complete test results - Update README.md: reorganize by recommendation priority - Document all 4 submission methods with pros/cons - Add troubleshooting for log visibility and resource limits - Simplify Quick Start to 2 commands (30 seconds) - Document Dask integration and resource optimization Covers kubectl, Jupyter, event-driven (AMQP), and Python CLI approaches.
Test validation proven by 93 passing tests, not narrative docs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Validates S1 GRD end-to-end workflows and improves workflow template for production use.
Changes
Tests (93 passing, 5.5s)
test_publish_amqp.py(parent.parent → parent.parent.parent)S1 GRD Workflow
workflows/run-s1-test.yamlexample for kubectl submissionDocumentation
Validated
✅ 93 tests pass
✅ S1 GRD converts successfully (~18-19min, 4 Dask workers)
✅ All 4 submission methods functional
✅ AMQP sensor running in production (devseed-staging)