LHP generate to databrick bundle - shim code feedback

[JJediny]

Forthmost, thank you for the great project

During our use of lhp we had to develop some shim scripts to faciliate our use of LHP -> Databricks Bundle -> Deployment. This project has been a great tool for us to maintain version control of our pipeline and perfect for our deployment otherwise. Here are some of the real use issues we're encountered.

These are the specific issues found in the scripts and the recommended fixes for the upstream LakehousePlumber project.

1. Python String Interpolation (Syntax Error)

• The Issue: LHP generates Python code with bare placeholder strings like database: "{catalog}.{schema}" inside decorators. This is not valid Python variable interpolation.
• The Shim Fix: fix_generated_pipelines.py uses regex to wrap these strings in f"...".
• Upstream Fix:
  • Update Template: The LHP Jinja2 templates for Python generation must produce f-strings (f"{...}") instead of bare strings.
  • Variable Scope: Ensure the generator explicitly outputs the configuration block (e.g., catalog = spark.conf.get(...)) at the top of the file so these f-strings have variables to reference.

2. Decorator Ordering (Runtime Error)

• The Issue: LHP generates code where expectation decorators (e.g., @dlt.expect) appear before the table decorator (e.g., @dlt.table). DLT requires the table/view decorator to be the outermost (top) decorator to register the dataset correctly.
• The Shim Fix: Regex logic swaps the order of these decorators.
• Upstream Fix:
  • Generator Logic: Adjust the generation loop to ensures the @dlt.table / @dlt.view decorator is always yielded first (at the top), followed by any expectations or access control decorators.

3. Import & API Standardization

• The Issue: LHP generates from pyspark import pipelines as dp and uses @dp.temporary_view.
  • The pipelines module is often deprecated or not available in standard DLT runtimes (which prefer import dlt).
  • temporary_view is not a standard DLT decorator (it's @dlt.view or just a function without a decorator for pure temporary scope).
• The Shim Fix: Replaces imports with import dlt as dp and maps temporary_view to view.
• Upstream Fix:
  • Modernize API: Update the generator to use the standard import dlt library.
  • Deprecate Custom Types: Remove LHP-specific types like temporary_view in favor of standard DLT view definitions.

4. Data Quality Visibility (temporary=True)

• The Issue: LHP defaults validation tables to temporary=True. In DLT, temporary tables do not record Expectation metrics to the event log, making Data Quality invisible in the DLT UI.
• The Shim Fix: Regex removes temporary=True from generated table definitions.
• Upstream Fix:
  • Configurable Default: Change the default for validation actions to temporary=False or expose this as a configurable YAML property (visible: true/false).

5. Spark Configuration Injection

• The Issue: The pipelines fail with AnalysisException without pipelines.incompatibleViewCheck.enabled="false". LHP provides no native way to inject top-level spark.conf.set calls into the generated Python file.
• The Shim Fix: Manually injects this line at the top of every file.
• Upstream Fix:
  • Schema Extension: Add a spark_configuration: section to the LHP YAML schema that generates corresponding spark.conf.set() calls in the Python output.

6. DLT Dependency Resolution

• The Issue: When using spark.sql(...) inside a DLT function, the lineage graph often fails to detect dependencies.
• The Shim Fix: Scans the SQL string for known table names and injects dummy dlt.read("...") calls to "hint" the dependency to DLT.
• Upstream Fix:
  • Explicit Reads: Instead of generating raw spark.sql(...), LHP should generate dlt.read("upstream_table") calls or explicit spark.readStream.table(...) references which DLT can track natively.

[Mmodarre]

Hi @JJediny — thanks for the detailed write-up and for using LHP! I've investigated each issue against both the current codebase and the latest Databricks Lakeflow Spark Declarative Pipelines documentation. Here's what I found:

---

1. Python String Interpolation — Not a bug

LHP resolves all substitutions (%{local_var}, {{ template }}, {env_token}, ${secret:...}) at generation time, not at Python runtime. The generated output contains fully-qualified string literals:

# Actual generated output (from E2E test baselines):
df = spark.readStream.table("acme_edw_dev.edw_raw.customer_raw")

There are no bare "{catalog}.{schema}" placeholders in generated Python files — the Jinja2 templates render already-resolved values via {{ full_table_name }}. The only case where f-strings appear in generated code is when ${secret:scope/key} placeholders require runtime resolution, which is handled by the SecretCodeGenerator.

If you're seeing unresolved {catalog} tokens in output, that indicates a substitution configuration issue (e.g., missing environment token definition in substitutions/), not a code generator bug. Happy to help debug your specific config if you can share a sanitized example.

---

2. Decorator Ordering — Valid for test actions only

You're right that the Databricks docs state: "Expectation decorators come after a @dp.table(), @dp.materialized_view or @dp.temporary_view() decorator."

For transforms and writes, LHP already generates the correct order. The Jinja2 templates place the table/view decorator outermost:

# From data_quality.py.j2 — correct order:
@dp.temporary_view()
@dp.expect_all_or_fail({...})
def target_view():

# From materialized_view.py.j2 — correct order:
@dp.materialized_view(...)
@dp.expect_all({...})
def table_name():

However, you found a real bug in the test generator (test_generator.py lines 147–159), which builds decorators programmatically and appends expectations before the @dp.table() decorator. I'll fix that — good catch.

---

3. Import & API Standardization — Not a bug (LHP uses the current standard)

The pyspark.pipelines module is the current recommended API, not a deprecated one. Per the official Databricks documentation:

"All Lakeflow Spark Declarative Pipelines Python APIs are implemented in the pyspark.pipelines module. Your pipeline code implemented with Python must explicitly import the pipelines module."

The old import dlt module has been replaced by pyspark.pipelines. Databricks explicitly recommends from pyspark import pipelines as dp.

Similarly, @dp.temporary_view() is the documented decorator for defining views:

"Databricks recommends using the pyspark.pipelines module (imported as dp) and the @temporary_view decorator."

Your shim replacing the import with import dlt as dp would revert to the older API. If your Databricks Runtime doesn't support pyspark.pipelines, you may need to update your DBR version (available since DBR 16.0+ / Spark 4.x).

---

4. Data Quality Visibility (temporary=True) — Partially valid

LHP defaults test actions to temporary=True since they're intermediate validation artifacts that typically shouldn't clutter the catalog. Regular streaming tables and materialized views default to temporary=False.

The Databricks docs describe a quarantine pattern that explicitly "combines expectations with temporary tables and views to track data quality metrics during pipeline updates", indicating that temporary objects do record expectation metrics.

That said, I can see the value in making this configurable. I'll consider adding a temporary property to test action YAML configuration so users can override the default. Could you share more details about what specifically becomes invisible in your DLT UI? That would help me understand if there's a nuance specific to @dp.table(temporary=True) vs @dp.temporary_view().

---

5. Spark Configuration Injection — Partially valid, but wrong solution suggested

LHP already supports spark_conf as a parameter on write targets (streaming tables and materialized views), which gets rendered as a decorator parameter — this is the correct DLT approach:

# Already supported in LHP YAML:
write_target:
  type: streaming_table
  spark_conf:
    "spark.databricks.delta.minFileSize": "134217728"

Generates:

dp.create_streaming_table(
    name="my_table",
    spark_conf={"spark.databricks.delta.minFileSize": "134217728"}
)

For pipelines.incompatibleViewCheck.enabled specifically — this is a pipeline-level configuration, not a per-file setting. The correct place to set it is in your Databricks Asset Bundle pipeline definition or pipeline settings:

# In your DAB bundle config:
resources:
  pipelines:
    my_pipeline:
      configuration:
        "pipelines.incompatibleViewCheck.enabled": "false"

Injecting spark.conf.set() at the top of generated Python files is a workaround, not the proper approach. I'll look into whether LHP's DAB integration (lhp generate -e dev -pc config/<my_pipelines_config.yaml>) should support propagating pipeline-level configuration.

---

6. DLT Dependency Resolution — Not a bug

LHP generates SQL transforms using stream(view_name) syntax, which is the documented DLT mechanism for declaring streaming dependencies in SQL:

# Actual LHP generated code:
@dp.temporary_view()
def v_orders_transformed():
    df = spark.sql("""SELECT ... FROM stream(v_orders_europe_raw)""")
    return df

stream() is a DLT-native SQL function that explicitly signals a streaming dependency. DLT parses SQL content to detect dependencies from both direct table references and stream() / live() / snapshot() calls. LHP also has its own SQLParser that extracts these references for build-time dependency ordering.

Injecting dummy dlt.read() calls that discard their results would generate incorrect code and could introduce side effects. If you're seeing lineage gaps in the DLT UI, it may be related to cross-pipeline references or external table references rather than intra-pipeline SQL dependencies.

---

Summary

#	Issue	Verdict	Action
1	String Interpolation	Not a bug	Likely a substitution config issue — happy to help debug
2	Decorator Ordering	Valid (test actions only)	Will fix test_generator.py
3	Import & API	Not a bug	pyspark.pipelines is the current Databricks standard
4	temporary=True DQ	Partially valid	Will consider making it configurable for test actions
5	Spark Config Injection	Partially valid	spark_conf already supported on write targets; pipeline-level config belongs in DAB
6	SQL Dependency	Not a bug	stream() is the documented dependency mechanism

Thanks again for the thorough report. It's clear you've put real effort into deploying LHP and I appreciate the feedback. The decorator ordering bug on test actions is a legitimate find that I'll address. For the other items, I hope the explanations above clarify LHP's design intent — and I'm happy to dig deeper on any of these if you have specific reproduction cases to share.

[Mmodarre]

@JJediny

Point 5 is implemented in version 0.7.5