Fixing MissingPlusPrefixDeprecation for job_cluster_config in dbt_project.yml causes “Missing required field” errors

[julie-aevean]

Describe the bug

Without + on job_cluster_config fields, dbt emits MissingPlusPrefixDeprecation warning(s).

After adding + to job_cluster_config fields to resolve the warning, Databricks job cluster creation fails with “Missing required field …” errors for all plus-prefixed fields.

Steps To Reproduce

1. In dbt_project.yml, configure Databricks job submission using submission_method: job_cluster and a
   job_cluster_config like:

Version A (produces deprecation warning):

models:
  project_name:
    +submission_method: job_cluster
    +job_cluster_config:
      spark_version: "16.4.x-photon-scala2.13"
      node_type_id: "md-fleet.xlarge"
      autoscale: {"min_workers": 1, "max_workers": 8}
      data_security_mode: "USER_ISOLATION"
      policy_id: "{{ env_var('DBT_POLICY_ID') }}"
      aws_attributes:
        availability: "ON_DEMAND"

2. Run dbt run -s model_name.
3. Observe MissingPlusPrefixDeprecation warnings referencing job_cluster_config fields.
4. Update dbt_project.yml to:

Version B (resolves deprecation warning, triggers runtime error):

models:
  project_name:
    +submission_method: job_cluster
    +job_cluster_config:
      +spark_version: "16.4.x-photon-scala2.13"
      +node_type_id: "md-fleet.xlarge"
      autoscale: {+"min_workers": 1, +"max_workers": 8}
      +data_security_mode: "USER_ISOLATION"
      +policy_id: "{{ env_var('DBT_POLICY_ID') }}"
      aws_attributes:
        +availability: "ON_DEMAND"

5. Run dbt run -s model_name again.
6. Observe the Databricks job cluster creation fails with errors like:

• Missing required field 'spark_version'
• Missing required field 'node_type_id'
• Missing required field 'autoscale'
• Missing required field 'data_security_mode'
• etc. (applies to all fields that were plus-prefixed)

Expected behavior

Either:

• job_cluster_config fields in dbt_project.yml do not require + prefixes (and do not trigger MissingPlusPrefixDeprecation), or
• if + prefixes are required to satisfy deprecation behavior, the resulting configuration should allow successful Databricks job cluster creation without the "Missing required field..." errors.

Screenshots and log output

• MissingPlusPrefixDeprecation warning screenshot/log excerpt:

• Error output showing “Missing required field …” after plus-prefixing:

System information

The output of dbt --version:

[asos-dipeshbhundia]

Seeing this same behaviour for contract enforcement, i've raised in dbt-core as well
dbt-labs/dbt-core#12349