Skip to content

Commit 74f97a5

Browse files
committed
engine: Add support for dynamic dependencies pt1
Add environment variable support to layer dependency processing. This means: Supporting ${VAR} syntax in X-Env-Layer-Requires Evaluating environment variables during layer discovery This is a trivial addition but a powerful feature. Update test harness and documentation. Fix hangover field name typo in docs, and regenerate.
1 parent a075ffd commit 74f97a5

File tree

10 files changed

+341
-11
lines changed

10 files changed

+341
-11
lines changed

docs/config/index.adoc

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -237,6 +237,49 @@ image:
237237
name: ${IGconf_device_class}-custom-image
238238
----
239239

240+
=== Environment Variable Dependencies
241+
242+
Layer dependencies can use environment variable expansion to create dynamic dependency resolution:
243+
244+
[source,yaml]
245+
----
246+
# METABEGIN
247+
# X-Env-Layer-Name: arch-specific-tools
248+
# X-Env-Layer-Requires: base-layer,${ARCH}-toolchain,${DISTRO}-packages
249+
# METAEND
250+
----
251+
252+
This allows layers to dynamically depend on other layers based on build-time environment variables:
253+
254+
[source,bash]
255+
----
256+
# Set environment variables before processing
257+
export ARCH=arm64
258+
export DISTRO=debian
259+
260+
# Layer dependencies will be equivalent to:
261+
# X-Env-Layer-Requires: base-layer, arm64-toolchain, debian-packages
262+
----
263+
264+
This feature enables dynamic layer selection. For example, one usage could be to use the same layer across different build environments to support toolchain selection based on target architecture. By using variables in the 'parent' layer, dynamic dependency resolution pulls in an architecture specific toolchain 'child' layer.
265+
266+
[NOTE]
267+
====
268+
Layer discovery will fail if variables referenced in dependencies are not set in the environment.
269+
====
270+
271+
[IMPORTANT]
272+
====
273+
Environment variable dependencies are evaluated during layer discovery, not during layer variable expansion. This ensures dependencies are resolved before layer processing begins. Because of this, only variables *already present* in the environment can be used in layer dependencies.
274+
275+
For example:
276+
277+
- System or custom environment variables (e.g., `USER`, `ARCH`, `DISTRO`)
278+
- Configuration file variables or command line overrides
279+
280+
All dependencies are always included, i.e. variable names result in actual layer names at build-time. For example, `${ARCH}-toolchain` always results in a dependency, but the specific layer name depends on the `ARCH` environment variable value.
281+
====
282+
240283
=== Expansion Context
241284

242285
During configuration processing, variables are expanded using:

docs/config/index.html

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -161,6 +161,7 @@
161161
<ul class="sectlevel2">
162162
<li><a href="#_environment_variable_expansion">Environment Variable Expansion</a></li>
163163
<li><a href="#_cross_reference_expansion">Cross-Reference Expansion</a></li>
164+
<li><a href="#_environment_variable_dependencies">Environment Variable Dependencies</a></li>
164165
<li><a href="#_expansion_context">Expansion Context</a></li>
165166
</ul>
166167
</li>
@@ -548,6 +549,80 @@ <h3 id="_cross_reference_expansion"><a class="anchor" href="#_cross_reference_ex
548549
</div>
549550
</div>
550551
<div class="sect2">
552+
<h3 id="_environment_variable_dependencies"><a class="anchor" href="#_environment_variable_dependencies"></a><a class="link" href="#_environment_variable_dependencies">Environment Variable Dependencies</a></h3>
553+
<div class="paragraph">
554+
<p>Layer dependencies can use environment variable expansion to create dynamic dependency resolution:</p>
555+
</div>
556+
<div class="listingblock">
557+
<div class="content">
558+
<pre class="highlight"><code class="language-yaml" data-lang="yaml"># METABEGIN
559+
# X-Env-Layer-Name: arch-specific-tools
560+
# X-Env-Layer-Requires: base-layer,${ARCH}-toolchain,${DISTRO}-packages
561+
# METAEND</code></pre>
562+
</div>
563+
</div>
564+
<div class="paragraph">
565+
<p>This allows layers to dynamically depend on other layers based on build-time environment variables:</p>
566+
</div>
567+
<div class="listingblock">
568+
<div class="content">
569+
<pre class="highlight"><code class="language-bash" data-lang="bash"># Set environment variables before processing
570+
export ARCH=arm64
571+
export DISTRO=debian
572+
573+
# Layer dependencies will be equivalent to:
574+
# X-Env-Layer-Requires: base-layer, arm64-toolchain, debian-packages</code></pre>
575+
</div>
576+
</div>
577+
<div class="paragraph">
578+
<p>This feature enables dynamic layer selection. For example, one usage could be to use the same layer across different build environments to support toolchain selection based on target architecture. By using variables in the 'parent' layer, dynamic dependency resolution pulls in an architecture specific toolchain 'child' layer.</p>
579+
</div>
580+
<div class="admonitionblock note">
581+
<table>
582+
<tr>
583+
<td class="icon">
584+
<div class="title">Note</div>
585+
</td>
586+
<td class="content">
587+
<div class="paragraph">
588+
<p>Layer discovery will fail if variables referenced in dependencies are not set in the environment.</p>
589+
</div>
590+
</td>
591+
</tr>
592+
</table>
593+
</div>
594+
<div class="admonitionblock important">
595+
<table>
596+
<tr>
597+
<td class="icon">
598+
<div class="title">Important</div>
599+
</td>
600+
<td class="content">
601+
<div class="paragraph">
602+
<p>Environment variable dependencies are evaluated during layer discovery, not during layer variable expansion. This ensures dependencies are resolved before layer processing begins. Because of this, only variables <strong>already present</strong> in the environment can be used in layer dependencies.</p>
603+
</div>
604+
<div class="paragraph">
605+
<p>For example:</p>
606+
</div>
607+
<div class="ulist">
608+
<ul>
609+
<li>
610+
<p>System or custom environment variables (e.g., <code>USER</code>, <code>ARCH</code>, <code>DISTRO</code>)</p>
611+
</li>
612+
<li>
613+
<p>Configuration file variables or command line overrides</p>
614+
</li>
615+
</ul>
616+
</div>
617+
<div class="paragraph">
618+
<p>All dependencies are always included, i.e. variable names result in actual layer names at build-time. For example, <code>${ARCH}-toolchain</code> always results in a dependency, but the specific layer name depends on the <code>ARCH</code> environment variable value.</p>
619+
</div>
620+
</td>
621+
</tr>
622+
</table>
623+
</div>
624+
</div>
625+
<div class="sect2">
551626
<h3 id="_expansion_context"><a class="anchor" href="#_expansion_context"></a><a class="link" href="#_expansion_context">Expansion Context</a></h3>
552627
<div class="paragraph">
553628
<p>During configuration processing, variables are expanded using:</p>

docs/index.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ X-Env metadata is contained within comment blocks:
4545
# X-Env-Layer-Name: This layer's name
4646
# X-Env-Layer-Description: Brief description of what this layer does
4747
# X-Env-Layer-Category: device
48-
# X-Env-Layer-Depends: base-layer,required-layer
48+
# X-Env-Layer-Requires: base-layer,required-layer
4949
# X-Env-VarPrefix: device
5050
# X-Env-Var-hostname: pi-${HOSTNAME_SUFFIX}
5151
# X-Env-Var-hostname-Description: System hostname for the device
@@ -66,13 +66,13 @@ mmdebstrap:
6666
- **`X-Env-Layer-Description`**: Human-readable description of the layer's purpose
6767
- **`X-Env-Layer-Category`**: Categorisation (device, image, etc.)
6868
- **`X-Env-Layer-Version`**: Version identifier for the layer
69-
- **`X-Env-Layer-Depends`**: Comma-separated list of required layers
69+
- **`X-Env-Layer-Requires`**: Comma-separated list of required layers
7070
- **`X-Env-Layer-Provides`**: Services or capabilities this layer provides
7171
- **`X-Env-Layer-RequiresProvider`**: Services or capabilities this layer requires
7272
- **`X-Env-Layer-Conflicts`**: Layers that cannot be used together with this one
7373

7474
### Dependencies and Providers
75-
**X-Env-Layer-Depends**
75+
**X-Env-Layer-Requires**
7676
- Direct layer references - "I need these specific layers"
7777
- Concrete dependencies - Must reference actual layer names
7878
- Build order enforcement - Dependencies are loaded first and are pull in automatically

docs/layer/index.adoc

Lines changed: 31 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ X-Env metadata is contained within comment blocks:
5050
# X-Env-Layer-Name: This layer's name
5151
# X-Env-Layer-Description: Brief description of what this layer does
5252
# X-Env-Layer-Category: device
53-
# X-Env-Layer-Depends: base-layer,required-layer
53+
# X-Env-Layer-Requires: base-layer,required-layer
5454
# X-Env-VarPrefix: device
5555
# X-Env-Var-hostname: pi-${HOSTNAME_SUFFIX}
5656
# X-Env-Var-hostname-Description: System hostname for the device
@@ -75,7 +75,7 @@ mmdebstrap:
7575

7676
`X-Env-Layer-Version`: Version identifier for the layer
7777

78-
`X-Env-Layer-Depends`: Comma-separated list of required layers
78+
`X-Env-Layer-Requires`: Comma-separated list of required layers
7979

8080
`X-Env-Layer-Provides`: Services or capabilities this layer provides
8181

@@ -84,13 +84,36 @@ mmdebstrap:
8484
`X-Env-Layer-Conflicts`: Layers that cannot be used together with this one
8585

8686
=== Dependencies and Providers
87-
==== X-Env-Layer-Depends
87+
88+
==== X-Env-Layer-Requires
8889

8990
- **Direct layer references**: "I need these specific layers"
9091
- **Concrete dependencies**: Must reference actual layer names
92+
- **Environment variable expansion**: Supports `${VAR}` syntax for dynamic dependencies
9193
- **Build order enforcement**: Dependencies are loaded first and are pull in automatically
9294
- **Example**: A device layer depends on a device base-layer because the base-layer provides mandatory settings inherited by the device layer.
9395

96+
[source,yaml]
97+
----
98+
# METABEGIN
99+
# X-Env-Layer-Name: conditional-layer
100+
# X-Env-Layer-Requires: base-layer,${ARCH}-toolchain,${DISTRO}-packages
101+
# METAEND
102+
----
103+
104+
Environment variables in dependencies are evaluated during layer discovery using the current environment context. If a required environment variable is not set, layer discovery will fail.
105+
106+
This enables dynamic layer dependency resolution based on build-time variables such as:
107+
108+
- **Architecture**: `${ARCH}-toolchain` resolves to `arm64-toolchain` when `ARCH=arm64`
109+
- **Distribution**: `${DISTRO}-packages` resolves to `debian-packages` when `DISTRO=debian`
110+
- **Build variant**: `${VARIANT}-config` for different build configurations
111+
112+
[IMPORTANT]
113+
====
114+
Only variables present in the environment can be used in dependencies.
115+
====
116+
94117
==== X-Env-Layer-Provides / X-Env-Layer-RequiresProvider
95118

96119
- **Abstract capability requirements**: "I need something that provides X"
@@ -99,6 +122,11 @@ mmdebstrap:
99122
- **Relationships**: If a provider is required, only one can exist in the overall configuration
100123
- **Example**: A layer requires a device provider, which could be satisfied by different device layers
101124

125+
[IMPORTANT]
126+
====
127+
Unlike dependencies, environment variables are not supported when evaluating providers.
128+
====
129+
102130
=== Environment Variables
103131
`X-Env-VarPrefix`: Prefix for all variables declared by this layer (e.g., `device`)
104132

docs/layer/index.html

Lines changed: 62 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -148,7 +148,7 @@
148148
<li><a href="#_layer_attributes">Layer Attributes</a></li>
149149
<li><a href="#_dependencies_and_providers">Dependencies and Providers</a>
150150
<ul class="sectlevel3">
151-
<li><a href="#_x_env_layer_depends">X-Env-Layer-Depends</a></li>
151+
<li><a href="#_x_env_layer_requires">X-Env-Layer-Requires</a></li>
152152
<li><a href="#_x_env_layer_provides_x_env_layer_requiresprovider">X-Env-Layer-Provides / X-Env-Layer-RequiresProvider</a></li>
153153
</ul>
154154
</li>
@@ -286,7 +286,7 @@ <h3 id="_metadata_structure"><a class="anchor" href="#_metadata_structure"></a><
286286
# X-Env-Layer-Name: This layer's name
287287
# X-Env-Layer-Description: Brief description of what this layer does
288288
# X-Env-Layer-Category: device
289-
# X-Env-Layer-Depends: base-layer,required-layer
289+
# X-Env-Layer-Requires: base-layer,required-layer
290290
# X-Env-VarPrefix: device
291291
# X-Env-Var-hostname: pi-${HOSTNAME_SUFFIX}
292292
# X-Env-Var-hostname-Description: System hostname for the device
@@ -318,7 +318,7 @@ <h3 id="_layer_attributes"><a class="anchor" href="#_layer_attributes"></a><a cl
318318
<p><code>X-Env-Layer-Version</code>: Version identifier for the layer</p>
319319
</div>
320320
<div class="paragraph">
321-
<p><code>X-Env-Layer-Depends</code>: Comma-separated list of required layers</p>
321+
<p><code>X-Env-Layer-Requires</code>: Comma-separated list of required layers</p>
322322
</div>
323323
<div class="paragraph">
324324
<p><code>X-Env-Layer-Provides</code>: Services or capabilities this layer provides</p>
@@ -333,7 +333,7 @@ <h3 id="_layer_attributes"><a class="anchor" href="#_layer_attributes"></a><a cl
333333
<div class="sect2">
334334
<h3 id="_dependencies_and_providers"><a class="anchor" href="#_dependencies_and_providers"></a><a class="link" href="#_dependencies_and_providers">Dependencies and Providers</a></h3>
335335
<div class="sect3">
336-
<h4 id="_x_env_layer_depends"><a class="anchor" href="#_x_env_layer_depends"></a><a class="link" href="#_x_env_layer_depends">X-Env-Layer-Depends</a></h4>
336+
<h4 id="_x_env_layer_requires"><a class="anchor" href="#_x_env_layer_requires"></a><a class="link" href="#_x_env_layer_requires">X-Env-Layer-Requires</a></h4>
337337
<div class="ulist">
338338
<ul>
339339
<li>
@@ -343,13 +343,57 @@ <h4 id="_x_env_layer_depends"><a class="anchor" href="#_x_env_layer_depends"></a
343343
<p><strong>Concrete dependencies</strong>: Must reference actual layer names</p>
344344
</li>
345345
<li>
346+
<p><strong>Environment variable expansion</strong>: Supports <code>${VAR}</code> syntax for dynamic dependencies</p>
347+
</li>
348+
<li>
346349
<p><strong>Build order enforcement</strong>: Dependencies are loaded first and are pull in automatically</p>
347350
</li>
348351
<li>
349352
<p><strong>Example</strong>: A device layer depends on a device base-layer because the base-layer provides mandatory settings inherited by the device layer.</p>
350353
</li>
351354
</ul>
352355
</div>
356+
<div class="listingblock">
357+
<div class="content">
358+
<pre class="highlight"><code class="language-yaml" data-lang="yaml"># METABEGIN
359+
# X-Env-Layer-Name: conditional-layer
360+
# X-Env-Layer-Requires: base-layer,${ARCH}-toolchain,${DISTRO}-packages
361+
# METAEND</code></pre>
362+
</div>
363+
</div>
364+
<div class="paragraph">
365+
<p>Environment variables in dependencies are evaluated during layer discovery using the current environment context. If a required environment variable is not set, layer discovery will fail.</p>
366+
</div>
367+
<div class="paragraph">
368+
<p>This enables dynamic layer dependency resolution based on build-time variables such as:</p>
369+
</div>
370+
<div class="ulist">
371+
<ul>
372+
<li>
373+
<p><strong>Architecture</strong>: <code>${ARCH}-toolchain</code> resolves to <code>arm64-toolchain</code> when <code>ARCH=arm64</code></p>
374+
</li>
375+
<li>
376+
<p><strong>Distribution</strong>: <code>${DISTRO}-packages</code> resolves to <code>debian-packages</code> when <code>DISTRO=debian</code></p>
377+
</li>
378+
<li>
379+
<p><strong>Build variant</strong>: <code>${VARIANT}-config</code> for different build configurations</p>
380+
</li>
381+
</ul>
382+
</div>
383+
<div class="admonitionblock important">
384+
<table>
385+
<tr>
386+
<td class="icon">
387+
<div class="title">Important</div>
388+
</td>
389+
<td class="content">
390+
<div class="paragraph">
391+
<p>Only variables present in the environment can be used in dependencies.</p>
392+
</div>
393+
</td>
394+
</tr>
395+
</table>
396+
</div>
353397
</div>
354398
<div class="sect3">
355399
<h4 id="_x_env_layer_provides_x_env_layer_requiresprovider"><a class="anchor" href="#_x_env_layer_provides_x_env_layer_requiresprovider"></a><a class="link" href="#_x_env_layer_provides_x_env_layer_requiresprovider">X-Env-Layer-Provides / X-Env-Layer-RequiresProvider</a></h4>
@@ -372,6 +416,20 @@ <h4 id="_x_env_layer_provides_x_env_layer_requiresprovider"><a class="anchor" hr
372416
</li>
373417
</ul>
374418
</div>
419+
<div class="admonitionblock important">
420+
<table>
421+
<tr>
422+
<td class="icon">
423+
<div class="title">Important</div>
424+
</td>
425+
<td class="content">
426+
<div class="paragraph">
427+
<p>Unlike dependencies, environment variables are not supported when evaluating providers.</p>
428+
</div>
429+
</td>
430+
</tr>
431+
</table>
432+
</div>
375433
</div>
376434
</div>
377435
<div class="sect2">

site/env_types.py

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -358,7 +358,7 @@ def from_metadata_fields(cls, metadata_dict: Dict[str, str],
358358

359359
@staticmethod
360360
def _parse_dependency_list(depends_str: str) -> List[str]:
361-
"""Parse dependency string into list of layer names/IDs."""
361+
"""Parse dependency string into list of layer names/IDs with environment variable evaluation."""
362362
if not depends_str.strip():
363363
return []
364364

@@ -367,6 +367,10 @@ def _parse_dependency_list(depends_str: str) -> List[str]:
367367
for dep in depends_str.split(','):
368368
dep_name = dep.strip()
369369
if dep_name:
370+
# Find and evaluate environment variables in dependency names
371+
if '${' in dep_name:
372+
dep_name = EnvLayer._evaluate_env_variables(dep_name)
373+
370374
# Validate dependency name format
371375
if re.search(r"\s", dep_name):
372376
raise ValueError(
@@ -376,6 +380,21 @@ def _parse_dependency_list(depends_str: str) -> List[str]:
376380
deps.append(dep_name)
377381
return deps
378382

383+
@staticmethod
384+
def _evaluate_env_variables(text: str) -> str:
385+
"""Evaluate ${VAR} environment variable substitutions in text."""
386+
import re
387+
import os
388+
389+
def replacer(match):
390+
var_name = match.group(1)
391+
env_value = os.environ.get(var_name)
392+
if env_value is None:
393+
raise ValueError(f"Environment variable '{var_name}' not found for dependency evaluation")
394+
return env_value
395+
396+
return re.sub(r'\$\{([A-Za-z_][A-Za-z0-9_]*)\}', replacer, text)
397+
379398
@classmethod
380399
def _validate_layer_fields(cls, metadata_dict: Dict[str, str], filepath: str = "") -> None:
381400
"""Validate that all X-Env-Layer fields are supported according to the schema"""

0 commit comments

Comments
 (0)