Support meltano container build [LFXV2-870]#20
Merged
Conversation
…oved build targets - Update .dockerignore with comprehensive glob patterns for better exclusions - Add separate Docker targets for v1-sync-helper and meltano containers - Create dedicated Dockerfile.meltano with optimized multi-stage build - Improve Makefile with distinct build and run targets for each service - Add help documentation for new Docker targets - Minor YAML formatting fixes in meltano.yml 🤖 Assisted with [GitHub Copilot](https://github.com/features/copilot) (via Zed) Signed-off-by: Eric Searcy <eric@linuxfoundation.org>
emsearcy
changed the title
- Create comprehensive development-focused documentation for AI agents - Document repository structure, build system, and container targets - Include new Makefile targets for container builds (v1-sync-helper, meltano) - Focus on codebase modification workflows rather than deployment procedures - Cover Go service architecture, Python ETL patterns, and CI/CD integration 🤖 Generated with [GitHub Copilot](https://github.com/features/copilot) (via Zed) Signed-off-by: Eric Searcy <eric@linuxfoundation.org>
- Rename ko-build-main.yaml to publish-main.yaml - Rename ko-build-tag.yaml to publish-release.yaml - Add new Meltano Docker build and release pipeline to publish-release.yaml - Implement sequential job dependencies: v1-sync-helper → meltano → helm-chart - Update AGENTS.md documentation to reflect new workflow structure - Maintain complete SLSA provenance and Cosign signing for all artifacts 🤖 Generated with [GitHub Copilot](https://github.com/features/copilot) (via Zed) Signed-off-by: Eric Searcy <eric@linuxfoundation.org>
Signed-off-by: Eric Searcy <eric@linuxfoundation.org>
Signed-off-by: Eric Searcy <eric@linuxfoundation.org>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR enhances the Docker containerization infrastructure by introducing a dedicated Meltano container with optimized multi-stage builds, reorganizes the build system with separate targets for each container type, and updates dependencies and configurations for improved consistency.
Key changes:
- Introduced new
docker/Dockerfile.meltanowith multi-stage build using uv for Python dependency management - Enhanced Makefile with separate build/run targets for v1-sync-helper and meltano containers
- Updated CI/CD workflows to build and publish both containers with proper dependency ordering and SLSA provenance
Reviewed changes
Copilot reviewed 16 out of 18 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
docker/Dockerfile.meltano |
New multi-stage Dockerfile for Meltano container with uv-based builds and non-root security |
meltano/Dockerfile |
Removed legacy Dockerfile in favor of centralized docker/ directory |
Makefile |
Added separate build targets for v1-sync-helper and meltano containers with comprehensive help documentation |
.dockerignore |
Enhanced with comprehensive glob patterns for better build context exclusion |
pyproject.toml |
Version bump to 0.1.1 and meltano dependency updated to 4.0.6 with s3 extras |
uv.lock |
Dependency updates including boto3/botocore additions for S3 support |
meltano/meltano.yml |
YAML formatting improvements for consistency (quotes, spacing) |
.github/workflows/publish-release.yaml |
Split into separate jobs for v1-sync-helper and meltano with provenance generation |
.github/workflows/publish-main.yaml |
Job renamed for clarity (publish-v1-sync-helper) |
.github/workflows/mega-linter.yml |
Updated MegaLinter version from 8.7.0 to 9.1.0 |
AGENTS.md |
New comprehensive documentation for AI agents/developers |
research/*.md |
Markdown table formatting improvements for consistency |
cmd/lfx-v1-sync-helper/README.md |
Table formatting improvements |
charts/lfx-v1-sync-helper/README.md |
Table formatting improvements |
.secretlintignore |
Added meltano plugin lock files to ignore list |
.lycheeignore |
Extended regex to include nats:// protocol |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Add shell script that generates a single Kubernetes ConfigMap containing catalogs from specified Meltano taps. Features: - Single ConfigMap with multiple keys (tap-name-catalog.json format) - Uses standard MELTANO_* environment variables - Displays current AWS account/profile and Meltano environment info - Captures meltano stderr to temp files for better error debugging - Validates JSON output and compacts for storage efficiency - Can output to file or stdout 🤖 Generated with GitHub Copilot (https://github.com/features/copilot) (via Zed) Signed-off-by: Eric Searcy <eric@linuxfoundation.org>
Split catalog generation back to individual ConfigMaps per tap instead of a single large ConfigMap to avoid Kubernetes size limits (1MB max). Also fix YAML document separator formatting issues. Changes: - Generate separate ConfigMaps (tap-dynamodb-catalog, tap-postgres-catalog) - Restore lfx.linuxfoundation.org/tap label for each ConfigMap - Fix YAML document separator (---) placement and indentation - Avoid ConfigMap size limit errors 🤖 Generated with GitHub Copilot (https://github.com/features/copilot) (via Zed) Signed-off-by: Eric Searcy <eric@linuxfoundation.org>
Add jq filtering to only include streams with selected=true in the generated catalog ConfigMaps. This dramatically reduces size by excluding ~4,969 unselected streams and keeping only ~616 selected streams, preventing Kubernetes ConfigMap size limit issues. 🤖 Generated with GitHub Copilot (https://github.com/features/copilot) (via Zed) Signed-off-by: Eric Searcy <eric@linuxfoundation.org>
jordane
approved these changes
Dec 18, 2025
bramwelt
reviewed
Dec 18, 2025
|
|
||
| # Run meltano Docker container | ||
| docker-run-meltano: docker-build-meltano | ||
| @echo "Summoning a dragon with the meltano Docker container..." |
There was a problem hiding this comment.
Oooh..Meltano's logo is a dragon. Got it.
Contributor
Author
There was a problem hiding this comment.
It prints one of several ascii-art dragons ... actually the GenAI output for this was to have it run "--help", but I changed it to be a little more fun. It's not reasonable to actually do anything with that container from a makefile, given it needs lots of different command line arguments to do different things ... but it does help work as a test that the app can "bootstrap".
Contributor
Author
There was a problem hiding this comment.
% uv run meltano --help | grep dragon
dragon Summon a dragon!
% uv run meltano dragon
/===-_---~~~~~~~~~------____
|===-~___ _,-'
-==\\ `//~\\ ~~~~`---.___.-~~
______-==| | | \\ _-~`
__--~~~ ,-/-==\\ | | `\ ,'
_-~ /' | \\ / / \ /
.' / | \\ /' / \ /'
/ ____ / | \`\.__/-~~ ~ \ _ _/' / \/'
/-'~ ~~~~~---__ | ~-/~ ( ) /' _--~`
\_| / _) ; ), __--~~
'~~--_/ _-~/- / \ '-~ \\
{\__--_/} / \\_>- )<__\ \\
/' (_/ _-~ | |__>--<__| |
|0 0 _/) )-~ | |__>--<__| |
/ /~ ,_/ / /__>---<__/ |
o o _// /-~_>---<__-~ /
(^(~ /~_>---<__- _-~
,/| /__>--<__/ _-~
,//('( |__>--<__| / .----_
( ( ')) |__>--<__| | /' _---_~\\
`-)) )) ( |__>--<__| | /' / ~\`\\
,/,'//( ( \__>--<__\ \ /' // ||
,( ( ((, )) ~-__>--<_~-_ ~--____---~' _/'/ /'
`~/ )` ) ,/| ~-_~>--<_/-__ __-~ _/
._-~//( )/ )) ` ~~-'_/_/ /~~~~~~~__--~
;'( ')/ ,)( ~~~~~~~~~~
' ') '( (/
' ' `
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR enhances the Docker configuration by introducing a dedicated meltano container and improving the overall build system organization.
Changes
Docker Infrastructure
docker/Dockerfile.meltanowith optimized multi-stage buildBuild System Improvements
docker-build-v1-sync-helper/docker-run-v1-sync-helperdocker-build-meltano/docker-run-meltanodocker-build-allfor building both containersConfiguration Updates
Technical Details
The new meltano container:
Testing
make docker-build-v1-sync-helpermake docker-build-meltanomake docker-build-allNotes
This is part of the effort to improve containerization and deployment flexibility for the LFX v1 sync helper system.
🤖 Assisted with GitHub Copilot (via Zed)