Add a new execution mode that preserves any undefined variable

preserve-undefined-execution-mode.md

@@ -0,0 +1,164 @@
+# PreserveUndefinedExecutionMode
+
+A new execution mode for Jinjava that preserves unknown/undefined variables as their original template syntax instead of rendering them as empty strings. This enables multi-pass rendering scenarios where templates are processed in stages with different variable contexts available at each stage.
+
+## Use Case
+
+Multi-pass template rendering is useful when:
+- Some variables are known at compile/build time (static values)
+- Other variables are only known at runtime (dynamic values)
+- You want to pre-render static parts while preserving dynamic placeholders
+
+## Usage
+
+```java
+import com.hubspot.jinjava.Jinjava;
+import com.hubspot.jinjava.JinjavaConfig;
+import com.hubspot.jinjava.mode.PreserveUndefinedExecutionMode;
+
+Jinjava jinjava = new Jinjava();
+JinjavaConfig config = JinjavaConfig.newBuilder()
+    .withExecutionMode(PreserveUndefinedExecutionMode.instance())
+    .build();
+
+Map<String, Object> context = new HashMap<>();
+context.put("staticValue", "STATIC");
+
+String template = "{{ staticValue }} - {{ dynamicValue }}";
+String result = jinjava.render(template, context, config);
+// Result: "STATIC - {{ dynamicValue }}"
+```
+
+## Behavior Summary
+
+| Feature | Input | Context | Output |
+|---------|-------|---------|--------|
+| Undefined expression | `{{ unknown }}` | `{}` | `{{ unknown }}` |
+| Defined expression | `{{ name }}` | `{name: "World"}` | `World` |
+| Expression with filter | `{{ name \| upper }}` | `{}` | `{{ name \| upper }}` |
+| Property access | `{{ obj.property }}` | `{}` | `{{ obj.property }}` |
+| Null value | `{{ nullVar }}` | `{nullVar: null}` | `{{ nullVar }}` |
+| Mixed | `Hello {{ name }}, {{ unknown }}!` | `{name: "World"}` | `Hello World, {{ unknown }}!` |
+
+### Control Structures
+
+| Feature | Input | Context | Output |
+|---------|-------|---------|--------|
+| If with known condition | `{% if true %}Hello{% endif %}` | `{}` | `Hello` |
+| If with unknown condition | `{% if unknown %}Hello{% endif %}` | `{}` | `{% if unknown %}Hello{% endif %}` |
+| If-else with unknown | `{% if unknown %}A{% else %}B{% endif %}` | `{}` | `{% if unknown %}A{% else %}B{% endif %}` |
+| For with known iterable | `{% for x in items %}{{ x }}{% endfor %}` | `{items: ["a","b"]}` | `ab` |
+| For with unknown iterable | `{% for x in items %}{{ x }}{% endfor %}` | `{}` | `{% for x in items %}{{ x }}{% endfor %}` |
+
+### Set Tags
+
+Set tags are preserved with their evaluated RHS values, enabling the variable to be set in subsequent rendering passes:
+
+| Feature | Input | Context | Output |
+|---------|-------|---------|--------|
+| Set with known RHS | `{% set x = name %}{{ x }}` | `{name: "World"}` | `{% set x = 'World' %}World` |
+| Set with unknown RHS | `{% set x = unknown %}{{ x }}` | `{}` | `{% set x = unknown %}{{ x }}` |
+
+### Comments
+
+Comment tags are preserved in output for multi-pass scenarios where comments may contain instructions for later processing stages:
+
+| Feature | Input | Context | Output |
+|---------|-------|---------|--------|
+| Simple comment | `{# this is a comment #}` | `{}` | `{# this is a comment #}` |
+| Inline comment | `Hello {# comment #} World` | `{}` | `Hello {# comment #} World` |
+| Comment with variables | `Hello {{ name }}{# comment #}!` | `{name: "World"}` | `Hello World{# comment #}!` |
+
+### Macros
+
+Macros are executed and their output is rendered, with only undefined variables within the macro output being preserved:
+
+```jinja
+{# macros.jinja #}
+{% macro greet(name) %}Hello {{ name }}, {{ title }}!{% endmacro %}
+```
+
+| Feature | Input | Context | Output |
+|---------|-------|---------|--------|
+| Macro with undefined var | `{{ m.greet('World') }}` | `{}` | `Hello World, {{ title }}!` |
+| Macro fully defined | `{{ m.greet('World') }}` | `{title: "Mr"}` | `Hello World, Mr!` |
+
+## Multi-Pass Rendering Example
+
+```java
+// First pass: render static values
+Map<String, Object> staticContext = new HashMap<>();
+staticContext.put("appName", "MyApp");
+staticContext.put("version", "1.0");
+
+JinjavaConfig preserveConfig = JinjavaConfig.newBuilder()
+    .withExecutionMode(PreserveUndefinedExecutionMode.instance())
+    .build();
+
+String template = "{{ appName }} v{{ version }} - Welcome {{ userName }}!";
+String firstPass = jinjava.render(template, staticContext, preserveConfig);
+// Result: "MyApp v1.0 - Welcome {{ userName }}!"
+
+// Second pass: render dynamic values
+Map<String, Object> dynamicContext = new HashMap<>();
+dynamicContext.put("userName", "Alice");
+
+JinjavaConfig defaultConfig = JinjavaConfig.newBuilder()
+    .withExecutionMode(DefaultExecutionMode.instance())
+    .build();
+
+String secondPass = jinjava.render(firstPass, dynamicContext, defaultConfig);
+// Result: "MyApp v1.0 - Welcome Alice!"
+```
+
+## Implementation Details
+
+`PreserveUndefinedExecutionMode` extends `EagerExecutionMode` and configures the context with:
+
+1. **PreserveUndefinedExpressionStrategy** - Returns original expression syntax when variables are undefined, instead of internal representations
+2. **DynamicVariableResolver** - Returns `DeferredValue.instance()` for undefined variables, triggering preservation
+3. **PartialMacroEvaluation** - Allows macros to execute and return partial results with undefined parts preserved
+4. **PreserveResolvedSetTags** - Preserves set tags even when RHS is fully resolved, enabling multi-pass variable binding
+5. **PreserveComments** - Outputs comment tags (`{# ... #}`) as-is instead of stripping them
+
+### New Context Flag: `isPreserveResolvedSetTags`
+
+A new context configuration flag was added to allow independent control over set tag preservation:
+
+```java
+// In ContextConfigurationIF
+default boolean isPreserveResolvedSetTags() {
+    return false;
+}
+
+// Usage in Context
+context.setPreserveResolvedSetTags(true);
+```
+
+This flag is checked in `EagerSetTagStrategy` to determine whether fully resolved set tags should be preserved in output or consumed during rendering.
+
+### Context Flag: `isPreserveComments`
+
+A context configuration flag to preserve comment tags in output:
+
+```java
+// In ContextConfigurationIF
+default boolean isPreserveComments() {
+    return false;
+}
+
+// Usage in Context
+context.setPreserveComments(true);
+```
+
+This flag is checked in `TreeParser` when processing note tokens. When enabled, comments are output as `TextNode` instead of being discarded.
+
+## Files Changed
+
+- `PreserveUndefinedExecutionMode.java` - Main execution mode implementation
+- `PreserveUndefinedExpressionStrategy.java` - Expression strategy for preserving original syntax
+- `ContextConfigurationIF.java` - Added `isPreserveResolvedSetTags` and `isPreserveComments` flags
+- `Context.java` - Added getter/setter for new flags
+- `EagerSetTagStrategy.java` - Modified to check `isPreserveResolvedSetTags` flag
+- `TreeParser.java` - Modified to check `isPreserveComments` flag
+- `PreserveUndefinedExecutionModeTest.java` - Test coverage

src/main/java/com/hubspot/jinjava/interpret/Context.java

@@ -875,6 +875,15 @@ public TemporaryValueClosable<Boolean> withPartialMacroEvaluation(
     return temporaryValueClosable;
   }
 
+  public boolean isPreserveResolvedSetTags() {
+    return contextConfiguration.isPreserveResolvedSetTags();
+  }
+
+  public void setPreserveResolvedSetTags(boolean preserveResolvedSetTags) {
+    contextConfiguration =
+      contextConfiguration.withPreserveResolvedSetTags(preserveResolvedSetTags);
+  }
+
   public boolean isUnwrapRawOverride() {
     return contextConfiguration.isUnwrapRawOverride();
   }
@@ -883,6 +892,22 @@ public void setUnwrapRawOverride(boolean unwrapRawOverride) {
     contextConfiguration = contextConfiguration.withUnwrapRawOverride(unwrapRawOverride);
   }
 
+  public boolean isPreserveComments() {
+    return contextConfiguration.isPreserveComments();
+  }
+
+  public void setPreserveComments(boolean preserveComments) {
+    contextConfiguration = contextConfiguration.withPreserveComments(preserveComments);
+  }
+
+  public boolean isExtendsDeferred() {
+    return contextConfiguration.isExtendsDeferred();
+  }
+
+  public void setExtendsDeferred(boolean extendsDeferred) {
+    contextConfiguration = contextConfiguration.withExtendsDeferred(extendsDeferred);
+  }
+
   public TemporaryValueClosable<Boolean> withUnwrapRawOverride() {
     TemporaryValueClosable<Boolean> temporaryValueClosable = new TemporaryValueClosable<>(
       isUnwrapRawOverride(),

src/main/java/com/hubspot/jinjava/interpret/ContextConfigurationIF.java

@@ -38,11 +38,26 @@ default boolean isPartialMacroEvaluation() {
     return false;
   }
 
+  @Default
+  default boolean isPreserveResolvedSetTags() {
+    return false;
+  }
+
   @Default
   default boolean isUnwrapRawOverride() {
     return false;
   }
 
+  @Default
+  default boolean isPreserveComments() {
+    return false;
+  }
+
+  @Default
+  default boolean isExtendsDeferred() {
+    return false;
+  }
+
   @Default
   default ErrorHandlingStrategy getErrorHandlingStrategy() {
     return ErrorHandlingStrategy.of();

src/main/java/com/hubspot/jinjava/lib/expression/PreserveUndefinedExpressionStrategy.java

@@ -0,0 +1,54 @@
+package com.hubspot.jinjava.lib.expression;
+
+import com.hubspot.jinjava.interpret.DeferredValueException;
+import com.hubspot.jinjava.interpret.JinjavaInterpreter;
+import com.hubspot.jinjava.lib.filter.EscapeFilter;
+import com.hubspot.jinjava.objects.SafeString;
+import com.hubspot.jinjava.tree.output.RenderedOutputNode;
+import com.hubspot.jinjava.tree.parse.ExpressionToken;
+import com.hubspot.jinjava.util.Logging;
+import org.apache.commons.lang3.StringUtils;
+
+public class PreserveUndefinedExpressionStrategy implements ExpressionStrategy {
+
+  private static final long serialVersionUID = 1L;
+
+  @Override
+  public RenderedOutputNode interpretOutput(
+    ExpressionToken master,
+    JinjavaInterpreter interpreter
+  ) {
+    Object var;
+    try {
+      var = interpreter.resolveELExpression(master.getExpr(), master.getLineNumber());
+    } catch (DeferredValueException e) {
+      return new RenderedOutputNode(master.getImage());
+    }
+
+    if (var == null) {
+      return new RenderedOutputNode(master.getImage());
+    }
+
+    String result = interpreter.getAsString(var);
+
+    if (interpreter.getConfig().isNestedInterpretationEnabled()) {
+      if (
+        !StringUtils.equals(result, master.getImage()) &&
+        (StringUtils.contains(result, master.getSymbols().getExpressionStart()) ||
+          StringUtils.contains(result, master.getSymbols().getExpressionStartWithTag()))
+      ) {
+        try {
+          result = interpreter.renderFlat(result);
+        } catch (Exception e) {
+          Logging.ENGINE_LOG.warn("Error rendering variable node result", e);
+        }
+      }
+    }
+
+    if (interpreter.getContext().isAutoEscape() && !(var instanceof SafeString)) {
+      result = EscapeFilter.escapeHtmlEntities(result);
+    }
+
+    return new RenderedOutputNode(result);
+  }
+}

src/main/java/com/hubspot/jinjava/lib/tag/BlockTag.java

@@ -20,6 +20,7 @@
 import com.hubspot.jinjava.doc.annotations.JinjavaParam;
 import com.hubspot.jinjava.doc.annotations.JinjavaSnippet;
 import com.hubspot.jinjava.doc.annotations.JinjavaTextMateSnippet;
+import com.hubspot.jinjava.interpret.DeferredValueException;
 import com.hubspot.jinjava.interpret.JinjavaInterpreter;
 import com.hubspot.jinjava.interpret.TemplateSyntaxException;
 import com.hubspot.jinjava.tree.TagNode;
@@ -60,6 +61,14 @@ public class BlockTag implements Tag {
 
   @Override
   public OutputNode interpretOutput(TagNode tagNode, JinjavaInterpreter interpreter) {
+    if (interpreter.getContext().isExtendsDeferred()) {
+      throw new DeferredValueException(
+        "block tag",
+        tagNode.getLineNumber(),
+        tagNode.getStartPosition()
+      );
+    }
+
     HelperStringTokenizer tagData = new HelperStringTokenizer(tagNode.getHelpers());
     if (!tagData.hasNext()) {
       throw new TemplateSyntaxException(

src/main/java/com/hubspot/jinjava/lib/tag/ExtendsTag.java

@@ -89,6 +89,7 @@ public class ExtendsTag implements Tag {
   @Override
   public String interpret(TagNode tagNode, JinjavaInterpreter interpreter) {
     if (interpreter.getContext().isDeferredExecutionMode()) {
+      interpreter.getContext().setExtendsDeferred(true);
       throw new DeferredValueException("extends tag");
     }
     HelperStringTokenizer tokenizer = new HelperStringTokenizer(tagNode.getHelpers());
@@ -101,11 +102,18 @@ public String interpret(TagNode tagNode, JinjavaInterpreter interpreter) {
       );
     }
 
-    String path = interpreter.resolveString(
-      tokenizer.next(),
-      tagNode.getLineNumber(),
-      tagNode.getStartPosition()
-    );
+    String path;
+    try {
+      path =
+        interpreter.resolveString(
+          tokenizer.next(),
+          tagNode.getLineNumber(),
+          tagNode.getStartPosition()
+        );
+    } catch (DeferredValueException e) {
+      interpreter.getContext().setExtendsDeferred(true);
+      throw e;
+    }
     path = interpreter.resolveResourceLocation(path);
     interpreter
       .getContext()

src/main/java/com/hubspot/jinjava/lib/tag/eager/EagerSetTagStrategy.java

@@ -58,6 +58,7 @@ public String run(TagNode tagNode, JinjavaInterpreter interpreter) {
     if (
       eagerExecutionResult.getResult().isFullyResolved() &&
       !interpreter.getContext().isDeferredExecutionMode() &&
+      !interpreter.getContext().isPreserveResolvedSetTags() &&
       (Arrays
           .stream(variables)
           .noneMatch(RelativePathResolver.CURRENT_PATH_CONTEXT_KEY::equals) ||