How to propagate the context into subsequent substitutions?

[ilg-ul]

I have an elaborate use case which can be simplified to something like this:

    "properties": {
      "valueConst": "const",
      "valueVariable": "{{ param }}"
    },
    "actions": {
      "one": "{% assign param = 'one' %}echo {{ param }} {{ properties.valueConst }} {{ properties.valueVariable }}",
      "two": "{% assign param = 'two' %}echo {{ param }} {{ properties.valueConst }} {{ properties.valueVariable }}"
    }

Please note two details:

• the substituted values may include other substitutions
• the value of param is assigned in each line (each action should behave like a function with parameters)

Since I could not find a configuration option for parseAndRender() to do the work in one step, I iterate until all substitutions are done:

    let current: string = input
    let substituted: string = current

    // Iterate until all substitutions are done.
    while (current.includes('{{') || current.includes('{%')) {
      // May throw.
      substituted = await this.engine.parseAndRender(current, map)

      if (substituted === current) {
        // If nothing changed, we're done.
        // Just a safety net, should not get there.
        break
      }
      current = substituted
    }

    return substituted

Unfortunately the context where param is stored is not passed to subsequent calls to parseAndRender().

Is there any configuration option to tell parseAndRender() to continue substituting until no more work, obviously using the same context?

If not, is there any way to split parseAndRender() into separate calls that use an explicit context, such that the context can be passed to subsequent calls?

Do you have any other suggestions how to achieve this functionality?

[jg-rp]

The second argument to parseAndRender() can be an instance of Context, which might do what you need.

import { Liquid, Context } from "liquidjs";

const engine = new Liquid();

const data = new Context({ param: "{{ foo }}", foo: 42 });
const template = `{{ param }}`;

const intermediate_result = await engine.parseAndRender(template, data);
console.log(intermediate_result);

engine.parseAndRender(intermediate_result, data).then(console.log);

If I've understood your requirements correctly, I can imagine a single pass solution that involves wrapping your data (strings that contain additional markup) in a custom drop class. If your drop class has a reference to an instance of Liquid and Context, it can render itself in place.

[ilg-ul]

Thank you for the quick response!

The second argument to parseAndRender() can be an instance of Context, which might do what you need.

Apparently it does.

I tried the following code:

import { Liquid, Context } from "liquidjs";

const engine = new Liquid();

const data = new Context({ valueWithParam: "the {{ param }} param", foo: 42 });

const template = `{% assign param = 'special' %}{{ valueWithParam }} is {{ foo }}`;

const template_result1 = await engine.parseAndRender(template, data);
console.log(template_result1);

const template_result2 = await engine.parseAndRender(template_result1, data);
console.log(template_result2);

The output was:

the {{ param }} param is 42
the special param is 42

So it looks like the assigned values are indeed stored in the context and can be used by subsequent substitutions.

I'll try this solution in my project, but I think that the question was answered.

[ilg-ul]

I can imagine a single pass solution that involves wrapping your data (strings that contain additional markup) in a custom drop class. If your drop class has a reference to an instance of Liquid and Context, it can render itself in place.

I am not familiar with Drops, so I cannot comment on this suggestion. But it seems complicated.

I thought on something much simpler, like a global boolean option passed to the engine so that it iterates internally until all substitutions are done.

[harttle]

For drops, I believe we can start with

    "properties": {
      "valueConst": "const",
      "valueVariable": new SomeDrop(engine, ctx, "{{ param }}")
    }

And in your drop:

class SomeDrop extends Drop {
  valueOf() {
    return engine.render(tpl, ctx)
  }
}

[ilg-ul]

Thank you, but in my use case the substitution values must follow the JSON syntax, and the implementation details must not be exposed to the user.

[ilg-ul]

I updated my code and I noticed that only assigned variables are passed via the context, the for variables are not.

In other words, something like this works:

  "properties": {
      "configs-ci": "native-cmake-clang,native-meson-clang,qemu-cortex-m0-cmake-gcc,qemu-cortex-m7f-cmake-gcc,qemu-riscv-rv32imac-cmake-gcc,qemu-cortex-m0-meson-gcc",
      "build-types": "debug,release",
      "test-actions": "prepare,build,test",
      "prepare-build-test-actions2": "{% assign test-actions = properties.test-actions | split: ',' %}{% for test-action in test-actions %}echo xpm run {{ test-action }} --config {{ configName }}{{ os.EOL }}{% endfor %}"
  },
  "actions": {
      "test-ci2": [
        "{% assign configs = properties.configs-ci | split: ',' %}{% assign build-types = properties.build-types | split: ',' %}{% for config in configs %}{% for build-type in build-types %}{% assign configName = config | append: '-' | append: build-type %}{{ properties.prepare-build-test-actions2 }}{% endfor %}{% endfor %}"
      ]
  }

but this does not work:

  "properties": {
      "configs-ci": "native-cmake-clang,native-meson-clang,qemu-cortex-m0-cmake-gcc,qemu-cortex-m7f-cmake-gcc,qemu-riscv-rv32imac-cmake-gcc,qemu-cortex-m0-meson-gcc",
      "build-types": "debug,release",
      "test-actions": "prepare,build,test",
      "prepare-build-test-actions3": "{% assign test-actions = properties.test-actions | split: ',' %}{% for test-action in test-actions %}echo xpm run {{ test-action }} --config {{ config }}-{{ build-type }}{{ os.EOL }}{% endfor %}"
  },
  "actions": {
      "test-ci3": [
        "{% assign configs = properties.configs-ci | split: ',' %}{% assign build-types = properties.build-types | split: ',' %}{% for config in configs %}{% for build-type in build-types %}{{ properties.prepare-build-test-actions3 }}{% endfor %}{% endfor %}"
      ]
  }

The difference is the variable configName where the two for variables are concatenated.

Note: In case you wonder what this might be and where is the markup, this is a configuration file where the values in the actions section are expanded using the properties above.

[harttle]

I think both won’t work but behave differently,

• variables created by assign will be overwritten by subsequent assign
• variables created by forloop will be popped after loop end

You need in place rendering. We can preprocess the input and wrap all values with a custom tag or drop. Then call parseandrender with current ctx in place.

[ilg-ul]

You are right, both won't work, for different reasons. :-(

[ilg-ul]

I think that I found a workaround.

Instead of passing each iteration value to the inner template, I pass the array and iterate in the inner template.

In this configuration there is only one assign per top template, so I avoid overriding the variable.

However it would be interesting to have a more generic solution.

[jg-rp]

How about something like this?

Note that I've removed the {% assign %} tags from your example JSON data.

import { Liquid, Drop } from "liquidjs";

class PropertiesDrop extends Drop {
  #derived_properties;
  #properties;
  #engine;
  #template_properties;

  constructor(engine, properties) {
    super();
    this.#engine = engine;
    this.#properties = properties;

    // Preprocess some properties.
    this.#derived_properties = new Map([
      ["configs-ci", properties["configs-ci"].split(",")],
      ["build-types", properties["build-types"].split(",")],
      ["test-actions", properties["test-actions"].split(",")],
    ]);

    // Names of properties that contain markup.
    this.#template_properties = new Set(["prepare-build-test-actions3"]);
  }

  async liquidMethodMissing(key, ctx) {
    if (this.#template_properties.has(key)) {
      // TODO: cache the parsed template so we parse it just the once.
      // `ctx` is available here since LiquidJS version 10.22.0.
      return await this.#engine.parseAndRender(this.#properties[key], ctx);
    }

    // Fall back to input properties
    return this.#derived_properties.get(key) || this.#properties[key];
  }
}

const engine = new Liquid();

const data = {
  properties: {
    "configs-ci":
      "native-cmake-clang,native-meson-clang,qemu-cortex-m0-cmake-gcc,qemu-cortex-m7f-cmake-gcc,qemu-riscv-rv32imac-cmake-gcc,qemu-cortex-m0-meson-gcc",
    "build-types": "debug,release",
    "test-actions": "prepare,build,test",
    "prepare-build-test-actions3":
      "{% for test-action in properties.test-actions %}echo xpm run {{ test-action }} --config {{ config }}-{{ build-type }}{{ os.EOL }}\n{% endfor %}",
  },
  actions: {
    "test-ci3": [
      "{% for config in properties.configs-ci %}{% for build-type in properties.build-types %}{{ properties.prepare-build-test-actions3 }}{% endfor %}{% endfor %}",
    ],
  },
};

// Could add `actions` here too if needed.
const ctx = { properties: new PropertiesDrop(engine, data.properties) };

for (const [action, template] of Object.entries(data.actions)) {
  const rendered = await engine.parseAndRender(template, ctx);
  console.log(rendered);
}

Output:

echo xpm run prepare --config native-cmake-clang-debug
echo xpm run build --config native-cmake-clang-debug
echo xpm run test --config native-cmake-clang-debug
echo xpm run prepare --config native-cmake-clang-release
echo xpm run build --config native-cmake-clang-release
echo xpm run test --config native-cmake-clang-release
echo xpm run prepare --config native-meson-clang-debug
echo xpm run build --config native-meson-clang-debug
echo xpm run test --config native-meson-clang-debug
echo xpm run prepare --config native-meson-clang-release
echo xpm run build --config native-meson-clang-release
echo xpm run test --config native-meson-clang-release
echo xpm run prepare --config qemu-cortex-m0-cmake-gcc-debug
echo xpm run build --config qemu-cortex-m0-cmake-gcc-debug
echo xpm run test --config qemu-cortex-m0-cmake-gcc-debug
echo xpm run prepare --config qemu-cortex-m0-cmake-gcc-release
echo xpm run build --config qemu-cortex-m0-cmake-gcc-release
echo xpm run test --config qemu-cortex-m0-cmake-gcc-release
echo xpm run prepare --config qemu-cortex-m7f-cmake-gcc-debug
echo xpm run build --config qemu-cortex-m7f-cmake-gcc-debug
echo xpm run test --config qemu-cortex-m7f-cmake-gcc-debug
echo xpm run prepare --config qemu-cortex-m7f-cmake-gcc-release
echo xpm run build --config qemu-cortex-m7f-cmake-gcc-release
echo xpm run test --config qemu-cortex-m7f-cmake-gcc-release
echo xpm run prepare --config qemu-riscv-rv32imac-cmake-gcc-debug
echo xpm run build --config qemu-riscv-rv32imac-cmake-gcc-debug
echo xpm run test --config qemu-riscv-rv32imac-cmake-gcc-debug
echo xpm run prepare --config qemu-riscv-rv32imac-cmake-gcc-release
echo xpm run build --config qemu-riscv-rv32imac-cmake-gcc-release
echo xpm run test --config qemu-riscv-rv32imac-cmake-gcc-release
echo xpm run prepare --config qemu-cortex-m0-meson-gcc-debug
echo xpm run build --config qemu-cortex-m0-meson-gcc-debug
echo xpm run test --config qemu-cortex-m0-meson-gcc-debug
echo xpm run prepare --config qemu-cortex-m0-meson-gcc-release
echo xpm run build --config qemu-cortex-m0-meson-gcc-release
echo xpm run test --config qemu-cortex-m0-meson-gcc-release

[ilg-ul]

Ok, so the Drops, amongst other features, are substituted immediately, which avoids the previously detected issues.

The problem for my use case (configuration files) is that I don't know in advance which properties in the configuration file will define templates that need to be substituted immediately, since they can be anywhere.

One possible solution would be to process all data below properties in the Drop.

If I understand your code right, for this case the liquidMethodMissing() method should parseAndRender() all properties that include {{ or {%, and return directly the other plain strings.

Would that be a reasonable approach?

[jg-rp]

If I understand your code right, for this case the liquidMethodMissing() method should parseAndRender() all properties that include {{ or {%, and return directly the other plain strings.

Would that be a reasonable approach?

Yes, quite reasonable.

liquidMethodMissing will get called a lot during template rendering, so it would be best to process properties early, outside liquidMethodMissing.

Also, if templates can be deeply nested within properties, the implementation of PropertiesDrop gets more complicated.

[ilg-ul]

I updated my tools and I think I have a functional solution.

Here is an example of actions dynamically generated from the above properties:

• https://github.com/micro-os-plus/utils-lists-xpack/blob/a0a677f55b5478ade2576e3cfe5bb98911383ac2/tests/package.json#L216-L233

Each of those entries generate several hundred commands, for all supported platforms, builders, compilers, build types, versions, etc.

The next step is to add some templates to automatically generate the next (countless) actions and build configuration.

Note: xpm actions are similar to npm scripts, but with some extra features (like using the liquid engine).