Skip to content

feat(common): Auto-generate complete-values-structure.yaml with objectname normalization#44955

Merged
Crow-Control merged 8 commits intocommon2026from
copilot/sub-pr-41017-again
Feb 14, 2026
Merged

feat(common): Auto-generate complete-values-structure.yaml with objectname normalization#44955
Crow-Control merged 8 commits intocommon2026from
copilot/sub-pr-41017-again

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Feb 14, 2026

Adds automated generation of complete-values-structure.yaml that aggregates all chart configurations into a normalized reference structure using placeholder values and objectname for variable-keyed nested objects.

Implementation

Script: generate_complete_values_structure.py

  • Collects 841+ values.yaml files from stable/incubator charts and common-test
  • Merges into comprehensive structure showing all possible configuration keys
  • Preserves existing comments via ruamel.yaml (falls back to PyYAML with warning)
  • Applies normalization post-merge

Normalization:

  • Variable-named keys → objectname placeholder
  • String values → ""
  • Large numbers → 0 (preserves 0-100 as likely config values)
  • Booleans → unchanged
  • Lists → first element only (structure example)

Parent keys with variable children:

PARENT_KEYS_WITH_VARIABLE_CHILDREN = {
    'workload', 'service', 'persistence', 'configmap', 'secret',
    'ingress', 'route', 'containers', 'initContainers', 'ports', 'hosts'
}

Result

File reduced from 14k to 6.5k lines. Example structure:

workload:
  objectname:  # replaces: main, backup, worker, etc.
    enabled: true
    type: ''
    podSpec:
      containers:
        objectname:  # replaces: main, sidecar, init, etc.
          enabled: true
          imageSelector: ''

service:
  objectname:  # replaces: main, api, metrics, etc.
    enabled: true
    ports:
      objectname:  # replaces: main, http, grpc, etc.
        port: 0
        protocol: ''

Workflow Integration

Added to schema-validation job in .github/workflows/common-tests.yaml:

  • Installs ruamel.yaml alongside PyYAML
  • Runs before test_schema.py
  • Comments persist across regenerations

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI and others added 3 commits February 14, 2026 21:07
…orkflow

Co-authored-by: PrivatePuffin <7613738+PrivatePuffin@users.noreply.github.com>
Co-authored-by: PrivatePuffin <7613738+PrivatePuffin@users.noreply.github.com>
Co-authored-by: PrivatePuffin <7613738+PrivatePuffin@users.noreply.github.com>
Copilot AI and others added 2 commits February 14, 2026 20:15
…ture.yaml

Co-authored-by: PrivatePuffin <7613738+PrivatePuffin@users.noreply.github.com>
Co-authored-by: PrivatePuffin <7613738+PrivatePuffin@users.noreply.github.com>
Copilot AI changed the title [WIP] Refactor common network policy and related changes feat(common): Auto-generate complete-values-structure.yaml with comment preservation Feb 14, 2026
Copilot AI and others added 3 commits February 14, 2026 20:22
…alues

Co-authored-by: PrivatePuffin <7613738+PrivatePuffin@users.noreply.github.com>
Co-authored-by: PrivatePuffin <7613738+PrivatePuffin@users.noreply.github.com>
…tname normalization

Co-authored-by: PrivatePuffin <7613738+PrivatePuffin@users.noreply.github.com>
Copilot AI changed the title feat(common): Auto-generate complete-values-structure.yaml with comment preservation feat(common): Auto-generate complete-values-structure.yaml with objectname normalization Feb 14, 2026
@Crow-Control Crow-Control marked this pull request as ready for review February 14, 2026 20:32
Copilot AI review requested due to automatic review settings February 14, 2026 20:32
@Crow-Control Crow-Control merged commit 203059c into common2026 Feb 14, 2026
2 checks passed
@Crow-Control Crow-Control deleted the copilot/sub-pr-41017-again branch February 14, 2026 20:32
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces an auto-generation workflow for charts/library/common/complete-values-structure.yaml, aggregating values from many charts and normalizing the output to a placeholder-based reference structure (notably using objectname for variable-keyed maps), and wires generation into CI before schema validation.

Changes:

  • Added generate_complete_values_structure.py to collect/merge many values.yaml sources, normalize variable keys, and emit a placeholder-based YAML.
  • Regenerated complete-values-structure.yaml with normalized placeholders and a generated-file header.
  • Updated .github/workflows/common-tests.yaml to install YAML tooling and run the generator before schema validation.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 7 comments.

File Description
charts/library/common/generate_complete_values_structure.py New generator that merges values, attempts to preserve comments, and normalizes output placeholders
charts/library/common/complete-values-structure.yaml Large regeneration of the reference structure with generated-file header and normalized placeholders
.github/workflows/common-tests.yaml CI now installs ruamel/PyYAML and runs the generator prior to schema validation

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +195 to +237
def collect_all_values_files(repo_root: Path) -> List[Path]:
"""Collect all values.yaml files from charts and common-test."""
values_files = []

# Collect from charts/stable/*
stable_dir = repo_root / "charts" / "stable"
if stable_dir.exists():
for chart_dir in stable_dir.iterdir():
if chart_dir.is_dir():
values_file = chart_dir / "values.yaml"
if values_file.exists():
values_files.append(values_file)

# Collect from charts/incubator/*
incubator_dir = repo_root / "charts" / "incubator"
if incubator_dir.exists():
for chart_dir in incubator_dir.iterdir():
if chart_dir.is_dir():
values_file = chart_dir / "values.yaml"
if values_file.exists():
values_files.append(values_file)

# Collect from common-test ci-values
common_test_ci_dir = repo_root / "charts" / "library" / "common-test" / "ci"
if common_test_ci_dir.exists():
for values_file in common_test_ci_dir.glob("*values.yaml"):
if values_file.is_file():
values_files.append(values_file)

# Add common-test main values files
common_test_dir = repo_root / "charts" / "library" / "common-test"
if common_test_dir.exists():
for name in ["values.yaml", "unit-values.yaml", "default-values.yaml"]:
values_file = common_test_dir / name
if values_file.exists():
values_files.append(values_file)

# Add common values.yaml (this should be processed first/last depending on priority)
common_values = repo_root / "charts" / "library" / "common" / "values.yaml"
if common_values.exists():
values_files.insert(0, common_values) # Add at beginning for base structure

return values_files
Copy link

Copilot AI Feb 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

collect_all_values_files() uses Path.iterdir() / glob() results without sorting. Because merge order determines insertion order (and normalize_variable_keys currently depends on first-key ordering), this can make the generated YAML unstable between runs/filesystems. Sort chart directories and collected paths to keep output deterministic.

Copilot uses AI. Check for mistakes.
Comment on lines +89 to +103
'workload', # workload.main, workload.backup, etc.
'service', # service.main, service.api, etc.
'persistence', # persistence.config, persistence.data, etc.
'configmap', # configmap.myconfig, configmap.settings, etc.
'secret', # secret.mysecret, secret.credentials, etc.
'ingress', # ingress.main, ingress.api, etc.
'route', # route.main, route.api, etc.
'containers', # containers.main, containers.sidecar, etc.
'initContainers',# initContainers.init, initContainers.setup, etc.
'ports', # ports.main, ports.http, ports.metrics, etc.
'hosts', # hosts.main, hosts.api, etc.
'middlewares', # middlewares.auth, middlewares.rate-limit, etc.
'rules', # Various rules with variable names
'backups', # backups.daily, backups.weekly, etc.
'pooler', # pooler.ro, pooler.rw, etc.
Copy link

Copilot AI Feb 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PARENT_KEYS_WITH_VARIABLE_CHILDREN is missing several top-level maps that are documented as variable-keyed (e.g. imagePullSecret.$name, serviceAccount.$name, rbac.$name, storageClass.$name, priorityClass.$name). As a result the generated structure keeps concrete example keys (like image-secret-name, sa-name, example1) instead of using the objectname placeholder. Add these parent keys to the set (or derive the set from schemas/docs) so normalization matches the documented value structure.

Suggested change
'workload', # workload.main, workload.backup, etc.
'service', # service.main, service.api, etc.
'persistence', # persistence.config, persistence.data, etc.
'configmap', # configmap.myconfig, configmap.settings, etc.
'secret', # secret.mysecret, secret.credentials, etc.
'ingress', # ingress.main, ingress.api, etc.
'route', # route.main, route.api, etc.
'containers', # containers.main, containers.sidecar, etc.
'initContainers',# initContainers.init, initContainers.setup, etc.
'ports', # ports.main, ports.http, ports.metrics, etc.
'hosts', # hosts.main, hosts.api, etc.
'middlewares', # middlewares.auth, middlewares.rate-limit, etc.
'rules', # Various rules with variable names
'backups', # backups.daily, backups.weekly, etc.
'pooler', # pooler.ro, pooler.rw, etc.
'workload', # workload.main, workload.backup, etc.
'service', # service.main, service.api, etc.
'persistence', # persistence.config, persistence.data, etc.
'configmap', # configmap.myconfig, configmap.settings, etc.
'secret', # secret.mysecret, secret.credentials, etc.
'ingress', # ingress.main, ingress.api, etc.
'route', # route.main, route.api, etc.
'containers', # containers.main, containers.sidecar, etc.
'initContainers', # initContainers.init, initContainers.setup, etc.
'ports', # ports.main, ports.http, ports.metrics, etc.
'hosts', # hosts.main, hosts.api, etc.
'middlewares', # middlewares.auth, middlewares.rate-limit, etc.
'rules', # Various rules with variable names
'backups', # backups.daily, backups.weekly, etc.
'pooler', # pooler.ro, pooler.rw, etc.
'imagePullSecret', # imagePullSecret.$name (e.g. image-secret-name)
'serviceAccount', # serviceAccount.$name (e.g. sa-name)
'rbac', # rbac.$name (e.g. example1)
'storageClass', # storageClass.$name
'priorityClass', # priorityClass.$name

Copilot uses AI. Check for mistakes.
Comment on lines +107 to +144
def normalize_value_to_placeholder(value: Any) -> Any:
"""
Convert actual values to appropriate placeholders.
- Strings become ""
- Numbers become 0 (or keep if likely a config value like port)
- Booleans stay as-is
- Lists: keep first element as example (shows structure)
- Dicts: retain structure with normalized values

Note: List normalization only preserves the first element pattern.
"""
if value is None:
return None
elif isinstance(value, bool):
return value # Keep booleans as-is
elif isinstance(value, str):
return "" # Always return empty string for string placeholders
elif isinstance(value, (int, float)):
# Keep small numbers that might be config values, zero out large ones
if isinstance(value, int) and 0 <= value <= 100:
return value # Likely a config value
return 0
elif isinstance(value, list):
if not value:
return []
# Keep first element as example (preserves structure pattern)
# Note: This shows the structure but doesn't preserve all list variations
return [normalize_value_to_placeholder(value[0])]
elif isinstance(value, dict):
# Keep structure but normalize all values
if HAS_RUAMEL:
from ruamel.yaml.comments import CommentedMap
result = CommentedMap() if isinstance(value, CommentedMap) else {}
else:
result = {}
for k, v in value.items():
result[k] = normalize_value_to_placeholder(v)
return result
Copy link

Copilot AI Feb 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The script/header claim comments will be preserved via ruamel.yaml, but both normalize_variable_keys() and normalize_value_to_placeholder() rebuild new maps without copying ruamel comment metadata. That means most existing inline comments will be lost even when ruamel is available. Either adjust the documentation/expectations, or implement normalization in-place (or explicitly copy .ca comment attributes) so comment preservation is real.

Copilot uses AI. Check for mistakes.
- name: Generate complete values structure
run: |
python3 charts/library/common/generate_complete_values_structure.py

Copy link

Copilot AI Feb 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The workflow regenerates complete-values-structure.yaml but never verifies the committed file matches the generated output. That can allow the repository to drift (CI passes while the checked-in reference file is stale). Consider adding a git diff --exit-code check after generation (or explicitly deciding to not track the generated file in git).

Suggested change
- name: Verify complete values structure is up-to-date
run: |
git diff --exit-code -- charts/library/common/complete-values-structure.yaml

Copilot uses AI. Check for mistakes.
Comment on lines +2911 to +2916
imagePullSecret:
image-secret-name:
enabled: true
# Kind: GRPCRoute, HTTPRoute, TCPRoute, TLSRoute, UDPRoute
kind: "HTTPRoute"
namespace: ""
labels: {}
annotations: {}

# Parent references
parentRefs:
- group: "gateway.networking.k8s.io"
kind: "Gateway"
name: "gateway-name"
namespace: "default"
sectionName: "https"

# Hostnames
hostnames:
- "app.example.com"

# Rules
rules:
- backendRefs:
- group: ""
kind: "Service"
name: "backend-service"
namespace: "default"
port: 8080
weight: 1
matches:
- path:
type: "PathPrefix"
value: "/"
headers:
- name: "X-Custom"
value: "value"
queryParams:
- name: "version"
value: "v1"
method: "GET"

# -----------------------------------------------------------------------------
# CONFIGMAPS
# -----------------------------------------------------------------------------
data:
Copy link

Copilot AI Feb 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This generated file is supposed to use an objectname placeholder for variable-keyed maps, but imagePullSecret is emitted with a concrete key (image-secret-name). That conflicts with the header note about objectname placeholders and with the documented imagePullSecret.$name structure; the generator should normalize this to objectname:.

Copilot uses AI. Check for mistakes.
Comment on lines 2942 to +2959
serviceAccount:
objectname:
main:
enabled: true
primary: true
namespace: ""
labels: {}
annotations: {}
targetSelectAll: false
targetSelectAll: true
sa-name:
enabled: true
primary: true
labels:
key: ''
key2: ''
annotations:
key: ''
key2: ''
other-sa-name:
enabled: true
targetSelector:
- "workload-name"

# -----------------------------------------------------------------------------
# RBAC
# -----------------------------------------------------------------------------
- ''
Copy link

Copilot AI Feb 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

serviceAccount is generated with concrete keys (main, sa-name, other-sa-name) instead of a single objectname placeholder, even though the values are documented under serviceAccount.$name. This suggests variable-key normalization is incomplete; consider regenerating with serviceAccount treated as a variable-key parent so the structure matches the docs.

Copilot uses AI. Check for mistakes.
Comment on lines +168 to +182

# Check if current parent_key is one that contains variable-named children
if parent_key in PARENT_KEYS_WITH_VARIABLE_CHILDREN:
# This dict contains variable-named objects
# Collect all the child objects and merge them into a single 'objectname' entry
if data:
# Get the first key as a template for the objectname entry
first_key = next(iter(data.keys()))
first_value = data[first_key]

# Recursively normalize the template value
normalized_template = normalize_variable_keys(first_value, first_key)

# Return a dict with just 'objectname' as the key
result['objectname'] = normalized_template
Copy link

Copilot AI Feb 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

normalize_variable_keys collapses variable-key maps to a single objectname entry by taking only the first child (first_key = next(iter(data.keys()))). This drops any keys that exist only in other named children (e.g., workload.main vs workload.worker), so the generated structure is not actually a union of all supported keys. Consider merging all child objects into the objectname template (e.g., reduce with merge_structures / merge_preserving_comments) before normalizing deeper.

Suggested change
# Check if current parent_key is one that contains variable-named children
if parent_key in PARENT_KEYS_WITH_VARIABLE_CHILDREN:
# This dict contains variable-named objects
# Collect all the child objects and merge them into a single 'objectname' entry
if data:
# Get the first key as a template for the objectname entry
first_key = next(iter(data.keys()))
first_value = data[first_key]
# Recursively normalize the template value
normalized_template = normalize_variable_keys(first_value, first_key)
# Return a dict with just 'objectname' as the key
result['objectname'] = normalized_template
# Local helper to deep-merge two mapping structures, forming a union of keys.
# Existing non-dict values in the target are preserved; nested dicts are merged recursively.
def _merge_structures(target: Any, source: Any) -> None:
if not isinstance(target, dict) or not isinstance(source, dict):
return
for k, v in source.items():
if k in target and isinstance(target[k], dict) and isinstance(v, dict):
_merge_structures(target[k], v)
elif k not in target:
target[k] = v
# Check if current parent_key is one that contains variable-named children
if parent_key in PARENT_KEYS_WITH_VARIABLE_CHILDREN:
# This dict contains variable-named objects
# Collect all the child objects and merge them into a single 'objectname' entry
if data:
# Build a merged template from all children so that we capture the union of keys
if HAS_RUAMEL:
from ruamel.yaml.comments import CommentedMap
merged_template = CommentedMap()
else:
merged_template = {}
for child_key, child_value in data.items():
# Recursively normalize each child value
normalized_child = normalize_variable_keys(child_value, child_key)
# Merge the normalized child into the accumulated template
_merge_structures(merged_template, normalized_child)
# Return a dict with just 'objectname' as the key
result['objectname'] = merged_template

Copilot uses AI. Check for mistakes.
Crow-Control added a commit that referenced this pull request Feb 14, 2026
…tname normalization (#44955)

Adds automated generation of `complete-values-structure.yaml` that
aggregates all chart configurations into a normalized reference
structure using placeholder values and `objectname` for variable-keyed
nested objects.

## Implementation

**Script:** `generate_complete_values_structure.py`
- Collects 841+ values.yaml files from stable/incubator charts and
common-test
- Merges into comprehensive structure showing all possible configuration
keys
- Preserves existing comments via ruamel.yaml (falls back to PyYAML with
warning)
- Applies normalization post-merge

**Normalization:**
- Variable-named keys → `objectname` placeholder
- String values → `""`
- Large numbers → `0` (preserves 0-100 as likely config values)
- Booleans → unchanged
- Lists → first element only (structure example)

**Parent keys with variable children:**
```python
PARENT_KEYS_WITH_VARIABLE_CHILDREN = {
    'workload', 'service', 'persistence', 'configmap', 'secret',
    'ingress', 'route', 'containers', 'initContainers', 'ports', 'hosts'
}
```

## Result

File reduced from 14k to 6.5k lines. Example structure:

```yaml
workload:
  objectname:  # replaces: main, backup, worker, etc.
    enabled: true
    type: ''
    podSpec:
      containers:
        objectname:  # replaces: main, sidecar, init, etc.
          enabled: true
          imageSelector: ''

service:
  objectname:  # replaces: main, api, metrics, etc.
    enabled: true
    ports:
      objectname:  # replaces: main, http, grpc, etc.
        port: 0
        protocol: ''
```

## Workflow Integration

Added to schema-validation job in `.github/workflows/common-tests.yaml`:
- Installs `ruamel.yaml` alongside PyYAML
- Runs before `test_schema.py`
- Comments persist across regenerations

<!-- START COPILOT CODING AGENT TIPS -->
---

✨ Let Copilot coding agent [set things up for
you](https://github.com/trueforge-org/truecharts/issues/new?title=✨+Set+up+Copilot+instructions&body=Configure%20instructions%20for%20this%20repository%20as%20documented%20in%20%5BBest%20practices%20for%20Copilot%20coding%20agent%20in%20your%20repository%5D%28https://gh.io/copilot-coding-agent-tips%29%2E%0A%0A%3COnboard%20this%20repo%3E&assignees=copilot)
— coding agent works faster and does higher quality work when set up for
your repo.

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: PrivatePuffin <7613738+PrivatePuffin@users.noreply.github.com>
Crow-Control added a commit that referenced this pull request Feb 15, 2026
…tname normalization (#44955)

Adds automated generation of `complete-values-structure.yaml` that
aggregates all chart configurations into a normalized reference
structure using placeholder values and `objectname` for variable-keyed
nested objects.

## Implementation

**Script:** `generate_complete_values_structure.py`
- Collects 841+ values.yaml files from stable/incubator charts and
common-test
- Merges into comprehensive structure showing all possible configuration
keys
- Preserves existing comments via ruamel.yaml (falls back to PyYAML with
warning)
- Applies normalization post-merge

**Normalization:**
- Variable-named keys → `objectname` placeholder
- String values → `""`
- Large numbers → `0` (preserves 0-100 as likely config values)
- Booleans → unchanged
- Lists → first element only (structure example)

**Parent keys with variable children:**
```python
PARENT_KEYS_WITH_VARIABLE_CHILDREN = {
    'workload', 'service', 'persistence', 'configmap', 'secret',
    'ingress', 'route', 'containers', 'initContainers', 'ports', 'hosts'
}
```

## Result

File reduced from 14k to 6.5k lines. Example structure:

```yaml
workload:
  objectname:  # replaces: main, backup, worker, etc.
    enabled: true
    type: ''
    podSpec:
      containers:
        objectname:  # replaces: main, sidecar, init, etc.
          enabled: true
          imageSelector: ''

service:
  objectname:  # replaces: main, api, metrics, etc.
    enabled: true
    ports:
      objectname:  # replaces: main, http, grpc, etc.
        port: 0
        protocol: ''
```

## Workflow Integration

Added to schema-validation job in `.github/workflows/common-tests.yaml`:
- Installs `ruamel.yaml` alongside PyYAML
- Runs before `test_schema.py`
- Comments persist across regenerations

<!-- START COPILOT CODING AGENT TIPS -->
---

✨ Let Copilot coding agent [set things up for
you](https://github.com/trueforge-org/truecharts/issues/new?title=✨+Set+up+Copilot+instructions&body=Configure%20instructions%20for%20this%20repository%20as%20documented%20in%20%5BBest%20practices%20for%20Copilot%20coding%20agent%20in%20your%20repository%5D%28https://gh.io/copilot-coding-agent-tips%29%2E%0A%0A%3COnboard%20this%20repo%3E&assignees=copilot)
— coding agent works faster and does higher quality work when set up for
your repo.

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: PrivatePuffin <7613738+PrivatePuffin@users.noreply.github.com>
@truecharts-admin
Copy link
Copy Markdown
Contributor

This PR is locked to prevent necro-posting on closed PRs. Please create a issue or contact staff on discord if you want to further discuss this

@trueforge-org trueforge-org locked as resolved and limited conversation to collaborators Feb 22, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants