feat(common): Auto-generate complete-values-structure.yaml with objectname normalization#44955
Conversation
…orkflow Co-authored-by: PrivatePuffin <7613738+PrivatePuffin@users.noreply.github.com>
Co-authored-by: PrivatePuffin <7613738+PrivatePuffin@users.noreply.github.com>
Co-authored-by: PrivatePuffin <7613738+PrivatePuffin@users.noreply.github.com>
7cd4314 to
c09247b
Compare
…ture.yaml Co-authored-by: PrivatePuffin <7613738+PrivatePuffin@users.noreply.github.com>
Co-authored-by: PrivatePuffin <7613738+PrivatePuffin@users.noreply.github.com>
…alues Co-authored-by: PrivatePuffin <7613738+PrivatePuffin@users.noreply.github.com>
Co-authored-by: PrivatePuffin <7613738+PrivatePuffin@users.noreply.github.com>
…tname normalization Co-authored-by: PrivatePuffin <7613738+PrivatePuffin@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR introduces an auto-generation workflow for charts/library/common/complete-values-structure.yaml, aggregating values from many charts and normalizing the output to a placeholder-based reference structure (notably using objectname for variable-keyed maps), and wires generation into CI before schema validation.
Changes:
- Added
generate_complete_values_structure.pyto collect/merge manyvalues.yamlsources, normalize variable keys, and emit a placeholder-based YAML. - Regenerated
complete-values-structure.yamlwith normalized placeholders and a generated-file header. - Updated
.github/workflows/common-tests.yamlto install YAML tooling and run the generator before schema validation.
Reviewed changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
| charts/library/common/generate_complete_values_structure.py | New generator that merges values, attempts to preserve comments, and normalizes output placeholders |
| charts/library/common/complete-values-structure.yaml | Large regeneration of the reference structure with generated-file header and normalized placeholders |
| .github/workflows/common-tests.yaml | CI now installs ruamel/PyYAML and runs the generator prior to schema validation |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| def collect_all_values_files(repo_root: Path) -> List[Path]: | ||
| """Collect all values.yaml files from charts and common-test.""" | ||
| values_files = [] | ||
|
|
||
| # Collect from charts/stable/* | ||
| stable_dir = repo_root / "charts" / "stable" | ||
| if stable_dir.exists(): | ||
| for chart_dir in stable_dir.iterdir(): | ||
| if chart_dir.is_dir(): | ||
| values_file = chart_dir / "values.yaml" | ||
| if values_file.exists(): | ||
| values_files.append(values_file) | ||
|
|
||
| # Collect from charts/incubator/* | ||
| incubator_dir = repo_root / "charts" / "incubator" | ||
| if incubator_dir.exists(): | ||
| for chart_dir in incubator_dir.iterdir(): | ||
| if chart_dir.is_dir(): | ||
| values_file = chart_dir / "values.yaml" | ||
| if values_file.exists(): | ||
| values_files.append(values_file) | ||
|
|
||
| # Collect from common-test ci-values | ||
| common_test_ci_dir = repo_root / "charts" / "library" / "common-test" / "ci" | ||
| if common_test_ci_dir.exists(): | ||
| for values_file in common_test_ci_dir.glob("*values.yaml"): | ||
| if values_file.is_file(): | ||
| values_files.append(values_file) | ||
|
|
||
| # Add common-test main values files | ||
| common_test_dir = repo_root / "charts" / "library" / "common-test" | ||
| if common_test_dir.exists(): | ||
| for name in ["values.yaml", "unit-values.yaml", "default-values.yaml"]: | ||
| values_file = common_test_dir / name | ||
| if values_file.exists(): | ||
| values_files.append(values_file) | ||
|
|
||
| # Add common values.yaml (this should be processed first/last depending on priority) | ||
| common_values = repo_root / "charts" / "library" / "common" / "values.yaml" | ||
| if common_values.exists(): | ||
| values_files.insert(0, common_values) # Add at beginning for base structure | ||
|
|
||
| return values_files |
There was a problem hiding this comment.
collect_all_values_files() uses Path.iterdir() / glob() results without sorting. Because merge order determines insertion order (and normalize_variable_keys currently depends on first-key ordering), this can make the generated YAML unstable between runs/filesystems. Sort chart directories and collected paths to keep output deterministic.
| 'workload', # workload.main, workload.backup, etc. | ||
| 'service', # service.main, service.api, etc. | ||
| 'persistence', # persistence.config, persistence.data, etc. | ||
| 'configmap', # configmap.myconfig, configmap.settings, etc. | ||
| 'secret', # secret.mysecret, secret.credentials, etc. | ||
| 'ingress', # ingress.main, ingress.api, etc. | ||
| 'route', # route.main, route.api, etc. | ||
| 'containers', # containers.main, containers.sidecar, etc. | ||
| 'initContainers',# initContainers.init, initContainers.setup, etc. | ||
| 'ports', # ports.main, ports.http, ports.metrics, etc. | ||
| 'hosts', # hosts.main, hosts.api, etc. | ||
| 'middlewares', # middlewares.auth, middlewares.rate-limit, etc. | ||
| 'rules', # Various rules with variable names | ||
| 'backups', # backups.daily, backups.weekly, etc. | ||
| 'pooler', # pooler.ro, pooler.rw, etc. |
There was a problem hiding this comment.
PARENT_KEYS_WITH_VARIABLE_CHILDREN is missing several top-level maps that are documented as variable-keyed (e.g. imagePullSecret.$name, serviceAccount.$name, rbac.$name, storageClass.$name, priorityClass.$name). As a result the generated structure keeps concrete example keys (like image-secret-name, sa-name, example1) instead of using the objectname placeholder. Add these parent keys to the set (or derive the set from schemas/docs) so normalization matches the documented value structure.
| 'workload', # workload.main, workload.backup, etc. | |
| 'service', # service.main, service.api, etc. | |
| 'persistence', # persistence.config, persistence.data, etc. | |
| 'configmap', # configmap.myconfig, configmap.settings, etc. | |
| 'secret', # secret.mysecret, secret.credentials, etc. | |
| 'ingress', # ingress.main, ingress.api, etc. | |
| 'route', # route.main, route.api, etc. | |
| 'containers', # containers.main, containers.sidecar, etc. | |
| 'initContainers',# initContainers.init, initContainers.setup, etc. | |
| 'ports', # ports.main, ports.http, ports.metrics, etc. | |
| 'hosts', # hosts.main, hosts.api, etc. | |
| 'middlewares', # middlewares.auth, middlewares.rate-limit, etc. | |
| 'rules', # Various rules with variable names | |
| 'backups', # backups.daily, backups.weekly, etc. | |
| 'pooler', # pooler.ro, pooler.rw, etc. | |
| 'workload', # workload.main, workload.backup, etc. | |
| 'service', # service.main, service.api, etc. | |
| 'persistence', # persistence.config, persistence.data, etc. | |
| 'configmap', # configmap.myconfig, configmap.settings, etc. | |
| 'secret', # secret.mysecret, secret.credentials, etc. | |
| 'ingress', # ingress.main, ingress.api, etc. | |
| 'route', # route.main, route.api, etc. | |
| 'containers', # containers.main, containers.sidecar, etc. | |
| 'initContainers', # initContainers.init, initContainers.setup, etc. | |
| 'ports', # ports.main, ports.http, ports.metrics, etc. | |
| 'hosts', # hosts.main, hosts.api, etc. | |
| 'middlewares', # middlewares.auth, middlewares.rate-limit, etc. | |
| 'rules', # Various rules with variable names | |
| 'backups', # backups.daily, backups.weekly, etc. | |
| 'pooler', # pooler.ro, pooler.rw, etc. | |
| 'imagePullSecret', # imagePullSecret.$name (e.g. image-secret-name) | |
| 'serviceAccount', # serviceAccount.$name (e.g. sa-name) | |
| 'rbac', # rbac.$name (e.g. example1) | |
| 'storageClass', # storageClass.$name | |
| 'priorityClass', # priorityClass.$name |
| def normalize_value_to_placeholder(value: Any) -> Any: | ||
| """ | ||
| Convert actual values to appropriate placeholders. | ||
| - Strings become "" | ||
| - Numbers become 0 (or keep if likely a config value like port) | ||
| - Booleans stay as-is | ||
| - Lists: keep first element as example (shows structure) | ||
| - Dicts: retain structure with normalized values | ||
|
|
||
| Note: List normalization only preserves the first element pattern. | ||
| """ | ||
| if value is None: | ||
| return None | ||
| elif isinstance(value, bool): | ||
| return value # Keep booleans as-is | ||
| elif isinstance(value, str): | ||
| return "" # Always return empty string for string placeholders | ||
| elif isinstance(value, (int, float)): | ||
| # Keep small numbers that might be config values, zero out large ones | ||
| if isinstance(value, int) and 0 <= value <= 100: | ||
| return value # Likely a config value | ||
| return 0 | ||
| elif isinstance(value, list): | ||
| if not value: | ||
| return [] | ||
| # Keep first element as example (preserves structure pattern) | ||
| # Note: This shows the structure but doesn't preserve all list variations | ||
| return [normalize_value_to_placeholder(value[0])] | ||
| elif isinstance(value, dict): | ||
| # Keep structure but normalize all values | ||
| if HAS_RUAMEL: | ||
| from ruamel.yaml.comments import CommentedMap | ||
| result = CommentedMap() if isinstance(value, CommentedMap) else {} | ||
| else: | ||
| result = {} | ||
| for k, v in value.items(): | ||
| result[k] = normalize_value_to_placeholder(v) | ||
| return result |
There was a problem hiding this comment.
The script/header claim comments will be preserved via ruamel.yaml, but both normalize_variable_keys() and normalize_value_to_placeholder() rebuild new maps without copying ruamel comment metadata. That means most existing inline comments will be lost even when ruamel is available. Either adjust the documentation/expectations, or implement normalization in-place (or explicitly copy .ca comment attributes) so comment preservation is real.
| - name: Generate complete values structure | ||
| run: | | ||
| python3 charts/library/common/generate_complete_values_structure.py | ||
|
|
There was a problem hiding this comment.
The workflow regenerates complete-values-structure.yaml but never verifies the committed file matches the generated output. That can allow the repository to drift (CI passes while the checked-in reference file is stale). Consider adding a git diff --exit-code check after generation (or explicitly deciding to not track the generated file in git).
| - name: Verify complete values structure is up-to-date | |
| run: | | |
| git diff --exit-code -- charts/library/common/complete-values-structure.yaml |
| imagePullSecret: | ||
| image-secret-name: | ||
| enabled: true | ||
| # Kind: GRPCRoute, HTTPRoute, TCPRoute, TLSRoute, UDPRoute | ||
| kind: "HTTPRoute" | ||
| namespace: "" | ||
| labels: {} | ||
| annotations: {} | ||
|
|
||
| # Parent references | ||
| parentRefs: | ||
| - group: "gateway.networking.k8s.io" | ||
| kind: "Gateway" | ||
| name: "gateway-name" | ||
| namespace: "default" | ||
| sectionName: "https" | ||
|
|
||
| # Hostnames | ||
| hostnames: | ||
| - "app.example.com" | ||
|
|
||
| # Rules | ||
| rules: | ||
| - backendRefs: | ||
| - group: "" | ||
| kind: "Service" | ||
| name: "backend-service" | ||
| namespace: "default" | ||
| port: 8080 | ||
| weight: 1 | ||
| matches: | ||
| - path: | ||
| type: "PathPrefix" | ||
| value: "/" | ||
| headers: | ||
| - name: "X-Custom" | ||
| value: "value" | ||
| queryParams: | ||
| - name: "version" | ||
| value: "v1" | ||
| method: "GET" | ||
|
|
||
| # ----------------------------------------------------------------------------- | ||
| # CONFIGMAPS | ||
| # ----------------------------------------------------------------------------- | ||
| data: |
There was a problem hiding this comment.
This generated file is supposed to use an objectname placeholder for variable-keyed maps, but imagePullSecret is emitted with a concrete key (image-secret-name). That conflicts with the header note about objectname placeholders and with the documented imagePullSecret.$name structure; the generator should normalize this to objectname:.
| serviceAccount: | ||
| objectname: | ||
| main: | ||
| enabled: true | ||
| primary: true | ||
| namespace: "" | ||
| labels: {} | ||
| annotations: {} | ||
| targetSelectAll: false | ||
| targetSelectAll: true | ||
| sa-name: | ||
| enabled: true | ||
| primary: true | ||
| labels: | ||
| key: '' | ||
| key2: '' | ||
| annotations: | ||
| key: '' | ||
| key2: '' | ||
| other-sa-name: | ||
| enabled: true | ||
| targetSelector: | ||
| - "workload-name" | ||
|
|
||
| # ----------------------------------------------------------------------------- | ||
| # RBAC | ||
| # ----------------------------------------------------------------------------- | ||
| - '' |
There was a problem hiding this comment.
serviceAccount is generated with concrete keys (main, sa-name, other-sa-name) instead of a single objectname placeholder, even though the values are documented under serviceAccount.$name. This suggests variable-key normalization is incomplete; consider regenerating with serviceAccount treated as a variable-key parent so the structure matches the docs.
|
|
||
| # Check if current parent_key is one that contains variable-named children | ||
| if parent_key in PARENT_KEYS_WITH_VARIABLE_CHILDREN: | ||
| # This dict contains variable-named objects | ||
| # Collect all the child objects and merge them into a single 'objectname' entry | ||
| if data: | ||
| # Get the first key as a template for the objectname entry | ||
| first_key = next(iter(data.keys())) | ||
| first_value = data[first_key] | ||
|
|
||
| # Recursively normalize the template value | ||
| normalized_template = normalize_variable_keys(first_value, first_key) | ||
|
|
||
| # Return a dict with just 'objectname' as the key | ||
| result['objectname'] = normalized_template |
There was a problem hiding this comment.
normalize_variable_keys collapses variable-key maps to a single objectname entry by taking only the first child (first_key = next(iter(data.keys()))). This drops any keys that exist only in other named children (e.g., workload.main vs workload.worker), so the generated structure is not actually a union of all supported keys. Consider merging all child objects into the objectname template (e.g., reduce with merge_structures / merge_preserving_comments) before normalizing deeper.
| # Check if current parent_key is one that contains variable-named children | |
| if parent_key in PARENT_KEYS_WITH_VARIABLE_CHILDREN: | |
| # This dict contains variable-named objects | |
| # Collect all the child objects and merge them into a single 'objectname' entry | |
| if data: | |
| # Get the first key as a template for the objectname entry | |
| first_key = next(iter(data.keys())) | |
| first_value = data[first_key] | |
| # Recursively normalize the template value | |
| normalized_template = normalize_variable_keys(first_value, first_key) | |
| # Return a dict with just 'objectname' as the key | |
| result['objectname'] = normalized_template | |
| # Local helper to deep-merge two mapping structures, forming a union of keys. | |
| # Existing non-dict values in the target are preserved; nested dicts are merged recursively. | |
| def _merge_structures(target: Any, source: Any) -> None: | |
| if not isinstance(target, dict) or not isinstance(source, dict): | |
| return | |
| for k, v in source.items(): | |
| if k in target and isinstance(target[k], dict) and isinstance(v, dict): | |
| _merge_structures(target[k], v) | |
| elif k not in target: | |
| target[k] = v | |
| # Check if current parent_key is one that contains variable-named children | |
| if parent_key in PARENT_KEYS_WITH_VARIABLE_CHILDREN: | |
| # This dict contains variable-named objects | |
| # Collect all the child objects and merge them into a single 'objectname' entry | |
| if data: | |
| # Build a merged template from all children so that we capture the union of keys | |
| if HAS_RUAMEL: | |
| from ruamel.yaml.comments import CommentedMap | |
| merged_template = CommentedMap() | |
| else: | |
| merged_template = {} | |
| for child_key, child_value in data.items(): | |
| # Recursively normalize each child value | |
| normalized_child = normalize_variable_keys(child_value, child_key) | |
| # Merge the normalized child into the accumulated template | |
| _merge_structures(merged_template, normalized_child) | |
| # Return a dict with just 'objectname' as the key | |
| result['objectname'] = merged_template |
…tname normalization (#44955) Adds automated generation of `complete-values-structure.yaml` that aggregates all chart configurations into a normalized reference structure using placeholder values and `objectname` for variable-keyed nested objects. ## Implementation **Script:** `generate_complete_values_structure.py` - Collects 841+ values.yaml files from stable/incubator charts and common-test - Merges into comprehensive structure showing all possible configuration keys - Preserves existing comments via ruamel.yaml (falls back to PyYAML with warning) - Applies normalization post-merge **Normalization:** - Variable-named keys → `objectname` placeholder - String values → `""` - Large numbers → `0` (preserves 0-100 as likely config values) - Booleans → unchanged - Lists → first element only (structure example) **Parent keys with variable children:** ```python PARENT_KEYS_WITH_VARIABLE_CHILDREN = { 'workload', 'service', 'persistence', 'configmap', 'secret', 'ingress', 'route', 'containers', 'initContainers', 'ports', 'hosts' } ``` ## Result File reduced from 14k to 6.5k lines. Example structure: ```yaml workload: objectname: # replaces: main, backup, worker, etc. enabled: true type: '' podSpec: containers: objectname: # replaces: main, sidecar, init, etc. enabled: true imageSelector: '' service: objectname: # replaces: main, api, metrics, etc. enabled: true ports: objectname: # replaces: main, http, grpc, etc. port: 0 protocol: '' ``` ## Workflow Integration Added to schema-validation job in `.github/workflows/common-tests.yaml`: - Installs `ruamel.yaml` alongside PyYAML - Runs before `test_schema.py` - Comments persist across regenerations <!-- START COPILOT CODING AGENT TIPS --> --- ✨ Let Copilot coding agent [set things up for you](https://github.com/trueforge-org/truecharts/issues/new?title=✨+Set+up+Copilot+instructions&body=Configure%20instructions%20for%20this%20repository%20as%20documented%20in%20%5BBest%20practices%20for%20Copilot%20coding%20agent%20in%20your%20repository%5D%28https://gh.io/copilot-coding-agent-tips%29%2E%0A%0A%3COnboard%20this%20repo%3E&assignees=copilot) — coding agent works faster and does higher quality work when set up for your repo. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: PrivatePuffin <7613738+PrivatePuffin@users.noreply.github.com>
…tname normalization (#44955) Adds automated generation of `complete-values-structure.yaml` that aggregates all chart configurations into a normalized reference structure using placeholder values and `objectname` for variable-keyed nested objects. ## Implementation **Script:** `generate_complete_values_structure.py` - Collects 841+ values.yaml files from stable/incubator charts and common-test - Merges into comprehensive structure showing all possible configuration keys - Preserves existing comments via ruamel.yaml (falls back to PyYAML with warning) - Applies normalization post-merge **Normalization:** - Variable-named keys → `objectname` placeholder - String values → `""` - Large numbers → `0` (preserves 0-100 as likely config values) - Booleans → unchanged - Lists → first element only (structure example) **Parent keys with variable children:** ```python PARENT_KEYS_WITH_VARIABLE_CHILDREN = { 'workload', 'service', 'persistence', 'configmap', 'secret', 'ingress', 'route', 'containers', 'initContainers', 'ports', 'hosts' } ``` ## Result File reduced from 14k to 6.5k lines. Example structure: ```yaml workload: objectname: # replaces: main, backup, worker, etc. enabled: true type: '' podSpec: containers: objectname: # replaces: main, sidecar, init, etc. enabled: true imageSelector: '' service: objectname: # replaces: main, api, metrics, etc. enabled: true ports: objectname: # replaces: main, http, grpc, etc. port: 0 protocol: '' ``` ## Workflow Integration Added to schema-validation job in `.github/workflows/common-tests.yaml`: - Installs `ruamel.yaml` alongside PyYAML - Runs before `test_schema.py` - Comments persist across regenerations <!-- START COPILOT CODING AGENT TIPS --> --- ✨ Let Copilot coding agent [set things up for you](https://github.com/trueforge-org/truecharts/issues/new?title=✨+Set+up+Copilot+instructions&body=Configure%20instructions%20for%20this%20repository%20as%20documented%20in%20%5BBest%20practices%20for%20Copilot%20coding%20agent%20in%20your%20repository%5D%28https://gh.io/copilot-coding-agent-tips%29%2E%0A%0A%3COnboard%20this%20repo%3E&assignees=copilot) — coding agent works faster and does higher quality work when set up for your repo. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: PrivatePuffin <7613738+PrivatePuffin@users.noreply.github.com>
|
This PR is locked to prevent necro-posting on closed PRs. Please create a issue or contact staff on discord if you want to further discuss this |
Adds automated generation of
complete-values-structure.yamlthat aggregates all chart configurations into a normalized reference structure using placeholder values andobjectnamefor variable-keyed nested objects.Implementation
Script:
generate_complete_values_structure.pyNormalization:
objectnameplaceholder""0(preserves 0-100 as likely config values)Parent keys with variable children:
Result
File reduced from 14k to 6.5k lines. Example structure:
Workflow Integration
Added to schema-validation job in
.github/workflows/common-tests.yaml:ruamel.yamlalongside PyYAMLtest_schema.py✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.