Enable references to workspace variables

[Skuvrik]

In short:

It should be possible to reference workspace variables in either the substitution files directly or via the databricks file.

Proposal:

In many projects, schemas in development has a unique prefix to enable parallel development.

To enable references to unique schemas for each developer, it should not be necessary to change variables manually in the substitution files.
It should be possible to make a reference directly to workspace variables in the substitution file i.e. ${workspace.current_user.id}.
Or, by referencing variables in the databricks file from the substitution file, i.e ${var.dev_prefix} in substitutions, and dev_prefix dev_${workspace.current_user.id} in the databricks file.

Example:

Referencing a variable in databricks.yml:
substitutions/dev.yaml

dev:
  raw_schema: ${var.dev_prefix}raw
  bronze_schema: ${var.dev_prefix}bronze

databricks.yml

variables:
   dev_prefix:
      description: "Prefix for development environment schemas"
      default: ''
targets:
   dev:
       variables:
         dev_prefix: dev_${workspace.current_user.id}_

OR

Referencing workspace variables directly:
substitutions/dev.yaml

dev:
  raw_schema: dev_${workspace.current_user.id}_raw
  bronze_schema: dev_${workspace.current_user.id}_bronze

Both result in

raw_schema = "dev_13829_raw"
bronze_schema = "dev_13829_bronze"

[Mmodarre]

@Skuvrik is this related to #57 ? Would you please have a look at that and let me know if we can merge the two ideas?

[Skuvrik]

@Skuvrik is this related to #57 ? Would you please have a look at that and let me know if we can merge the two ideas?

TLDR; No :)

I think the goal for these two requests are close to the same - to enable parallel development in Databricks.

The difference is in the approach. In the #57 suggestion, a separate substitution file is made for each feature to allow separate deployments. In this suggestion, a single substitution file will change for each user .

I think the #57 suggestion does not need any development from the lhp side. The suggested state can be achieved relatively easily by creating a "feature/integration" target in the Databricks file and commenting out all but one substitution file according to what one is working on.

But that will create an overhead as they need to create separate substitution files for each integration with custom schema names. These files are not suitable for living in a project repo (think of the scaling issues and need for clean-ups), and thus also escape standardizations and reviews.

Enabling reference to workspace variables would make it possible to enable personalized deployments for each developer by just having a single substitution file for all. It might not be the same as the other suggestion, but it will achieve the same goal of parallel development without the overhead.

[Mmodarre]

@Skuvrik
The "Referencing workspace variables directly" approach cannot work as described because:

1. Databricks bundle variables are resolved at deploy time, not runtime
   2.Generated Python code contains literal strings - when a flowgroup uses database: {schema} and schema: dev_${workspace.current_user.id}_raw, the generated code becomes:

   dp.create_streaming_table(name="dev_${workspace.current_user.id}_raw.table")

DLT runtime sees literal strings - the Python file has already been generated, and there's no mechanism to resolve bundle variables in Python code